Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250111 02:56:58.384343 18513 test_util.cc:274] Using random seed: 1566429004
2025-01-11T02:56:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-11T02:56:58Z Disabled control of system clock
I20250111 02:56:58.458393 18513 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
/tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.126:45115
--webserver_interface=127.18.20.126
--webserver_port=0
--builtin_ntp_servers=127.18.20.84:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.20.126:45115,127.18.20.125:40359,127.18.20.124:34593
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250111 02:56:58.770310 18522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:56:58.770964 18522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:56:58.771497 18522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:56:58.803656 18522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 02:56:58.804046 18522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:56:58.804332 18522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 02:56:58.804589 18522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 02:56:58.842437 18522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.20.84:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.20.126:45115,127.18.20.125:40359,127.18.20.124:34593
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.126:45115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.20.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4120
TSAN enabled
I20250111 02:56:58.843972 18522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:56:58.845743 18522 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:56:58.863025 18528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:56:58.864037 18529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:56:58.865775 18531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:56:58.866616 18522 server_base.cc:1034] running on GCE node
I20250111 02:57:00.051460 18522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:57:00.054137 18522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:57:00.055523 18522 hybrid_clock.cc:648] HybridClock initialized: now 1736564220055479 us; error 60 us; skew 500 ppm
I20250111 02:57:00.056360 18522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:57:00.067991 18522 webserver.cc:458] Webserver started at http://127.18.20.126:36887/ using document root <none> and password file <none>
I20250111 02:57:00.069025 18522 fs_manager.cc:362] Metadata directory not provided
I20250111 02:57:00.069267 18522 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:57:00.069718 18522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:57:00.074967 18522 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/data/instance:
uuid: "1e336d4ddf044bf5b0b6e4f5af634727"
format_stamp: "Formatted at 2025-01-11 02:57:00 on dist-test-slave-09x0"
I20250111 02:57:00.076329 18522 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/wal/instance:
uuid: "1e336d4ddf044bf5b0b6e4f5af634727"
format_stamp: "Formatted at 2025-01-11 02:57:00 on dist-test-slave-09x0"
I20250111 02:57:00.084271 18522 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250111 02:57:00.090442 18538 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:57:00.091717 18522 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250111 02:57:00.092051 18522 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/data,/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/wal
uuid: "1e336d4ddf044bf5b0b6e4f5af634727"
format_stamp: "Formatted at 2025-01-11 02:57:00 on dist-test-slave-09x0"
I20250111 02:57:00.092397 18522 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:57:00.159709 18522 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:57:00.161257 18522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:57:00.161720 18522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:57:00.235401 18522 rpc_server.cc:307] RPC server started. Bound to: 127.18.20.126:45115
I20250111 02:57:00.235497 18589 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.20.126:45115 every 8 connection(s)
I20250111 02:57:00.238241 18522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/data/info.pb
I20250111 02:57:00.240244 18513 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu as pid 18522
I20250111 02:57:00.240711 18513 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/wal/instance
I20250111 02:57:00.244863 18590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:00.245278 18513 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
/tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.125:40359
--webserver_interface=127.18.20.125
--webserver_port=0
--builtin_ntp_servers=127.18.20.84:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.20.126:45115,127.18.20.125:40359,127.18.20.124:34593
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250111 02:57:00.256652 18590 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:57:00.282655 18590 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } has no permanent_uuid. Determining permanent_uuid...
W20250111 02:57:00.285310 18539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:40359: connect: Connection refused (error 111)
W20250111 02:57:00.289799 18590 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.125:40359: Network error: Client connection negotiation failed: client connection to 127.18.20.125:40359: connect: Connection refused (error 111)
I20250111 02:57:00.339433 18590 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } attempt: 1
W20250111 02:57:00.343637 18590 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.125:40359: Network error: Client connection negotiation failed: client connection to 127.18.20.125:40359: connect: Connection refused (error 111)
I20250111 02:57:00.412195 18590 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } attempt: 2
W20250111 02:57:00.416437 18590 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.125:40359: Network error: Client connection negotiation failed: client connection to 127.18.20.125:40359: connect: Connection refused (error 111)
I20250111 02:57:00.508006 18590 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } attempt: 3
W20250111 02:57:00.512140 18590 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.125:40359: Network error: Client connection negotiation failed: client connection to 127.18.20.125:40359: connect: Connection refused (error 111)
W20250111 02:57:00.570289 18592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:57:00.570873 18592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:57:00.571377 18592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:57:00.603545 18592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 02:57:00.603834 18592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:57:00.604038 18592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 02:57:00.604234 18592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 02:57:00.640558 18592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.20.84:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.20.126:45115,127.18.20.125:40359,127.18.20.124:34593
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.125:40359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.20.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4120
TSAN enabled
I20250111 02:57:00.641935 18592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:57:00.643719 18592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 02:57:00.655616 18590 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } attempt: 4
W20250111 02:57:00.658800 18601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:57:00.662319 18590 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.125:40359: Network error: Client connection negotiation failed: client connection to 127.18.20.125:40359: connect: Connection refused (error 111)
I20250111 02:57:00.961738 18590 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } attempt: 5
W20250111 02:57:00.967823 18590 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.125:40359: Network error: Client connection negotiation failed: client connection to 127.18.20.125:40359: connect: Connection refused (error 111)
I20250111 02:57:01.515216 18590 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } attempt: 6
W20250111 02:57:01.526248 18590 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.125:40359: Network error: Client connection negotiation failed: client connection to 127.18.20.125:40359: connect: Connection refused (error 111)
W20250111 02:57:00.662053 18602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:00.663268 18592 server_base.cc:1034] running on GCE node
W20250111 02:57:00.664896 18604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:01.856683 18592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:57:01.859644 18592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:57:01.861092 18592 hybrid_clock.cc:648] HybridClock initialized: now 1736564221861037 us; error 69 us; skew 500 ppm
I20250111 02:57:01.862026 18592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:57:01.869190 18592 webserver.cc:458] Webserver started at http://127.18.20.125:39483/ using document root <none> and password file <none>
I20250111 02:57:01.870209 18592 fs_manager.cc:362] Metadata directory not provided
I20250111 02:57:01.870433 18592 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:57:01.870903 18592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:57:01.876129 18592 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/data/instance:
uuid: "8784b9603f8544b79373158aace063af"
format_stamp: "Formatted at 2025-01-11 02:57:01 on dist-test-slave-09x0"
I20250111 02:57:01.877388 18592 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/wal/instance:
uuid: "8784b9603f8544b79373158aace063af"
format_stamp: "Formatted at 2025-01-11 02:57:01 on dist-test-slave-09x0"
I20250111 02:57:01.885218 18592 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250111 02:57:01.891259 18613 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:57:01.892444 18592 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250111 02:57:01.892799 18592 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/data,/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/wal
uuid: "8784b9603f8544b79373158aace063af"
format_stamp: "Formatted at 2025-01-11 02:57:01 on dist-test-slave-09x0"
I20250111 02:57:01.893162 18592 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:57:01.947052 18592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:57:01.948673 18592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:57:01.949177 18592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:57:02.021469 18592 rpc_server.cc:307] RPC server started. Bound to: 127.18.20.125:40359
I20250111 02:57:02.021559 18664 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.20.125:40359 every 8 connection(s)
I20250111 02:57:02.024354 18592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/data/info.pb
I20250111 02:57:02.025830 18513 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu as pid 18592
I20250111 02:57:02.026405 18513 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-1/wal/instance
I20250111 02:57:02.029740 18513 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
/tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.124:34593
--webserver_interface=127.18.20.124
--webserver_port=0
--builtin_ntp_servers=127.18.20.84:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.20.126:45115,127.18.20.125:40359,127.18.20.124:34593
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250111 02:57:02.031309 18665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:02.042992 18665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:57:02.069322 18665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:57:02.084498 18665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } has no permanent_uuid. Determining permanent_uuid...
W20250111 02:57:02.086983 18615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.124:34593: connect: Connection refused (error 111)
W20250111 02:57:02.091310 18665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:34593: Network error: Client connection negotiation failed: client connection to 127.18.20.124:34593: connect: Connection refused (error 111)
I20250111 02:57:02.140946 18665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } attempt: 1
W20250111 02:57:02.145080 18665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:34593: Network error: Client connection negotiation failed: client connection to 127.18.20.124:34593: connect: Connection refused (error 111)
I20250111 02:57:02.213672 18665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } attempt: 2
W20250111 02:57:02.217815 18665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:34593: Network error: Client connection negotiation failed: client connection to 127.18.20.124:34593: connect: Connection refused (error 111)
I20250111 02:57:02.309366 18665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } attempt: 3
W20250111 02:57:02.313457 18665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:34593: Network error: Client connection negotiation failed: client connection to 127.18.20.124:34593: connect: Connection refused (error 111)
W20250111 02:57:02.355397 18667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:57:02.355959 18667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:57:02.356410 18667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:57:02.388199 18667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 02:57:02.388494 18667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:57:02.388705 18667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 02:57:02.388902 18667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 02:57:02.425275 18667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.20.84:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.20.126:45115,127.18.20.125:40359,127.18.20.124:34593
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.124:34593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.20.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4120
TSAN enabled
I20250111 02:57:02.426682 18667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:57:02.428479 18667 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:57:02.449688 18677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:02.456945 18665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } attempt: 4
W20250111 02:57:02.461407 18665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:34593: Network error: Client connection negotiation failed: client connection to 127.18.20.124:34593: connect: Connection refused (error 111)
I20250111 02:57:02.586951 18590 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } attempt: 7
I20250111 02:57:02.599956 18590 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } has no permanent_uuid. Determining permanent_uuid...
W20250111 02:57:02.605695 18590 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:34593: Network error: Client connection negotiation failed: client connection to 127.18.20.124:34593: connect: Connection refused (error 111)
I20250111 02:57:02.664412 18590 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } attempt: 1
W20250111 02:57:02.668865 18590 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:34593: Network error: Client connection negotiation failed: client connection to 127.18.20.124:34593: connect: Connection refused (error 111)
I20250111 02:57:02.750602 18590 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } attempt: 2
W20250111 02:57:02.756079 18590 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:34593: Network error: Client connection negotiation failed: client connection to 127.18.20.124:34593: connect: Connection refused (error 111)
I20250111 02:57:02.760843 18665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } attempt: 5
W20250111 02:57:02.765298 18665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:34593: Network error: Client connection negotiation failed: client connection to 127.18.20.124:34593: connect: Connection refused (error 111)
I20250111 02:57:02.841795 18590 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } attempt: 3
W20250111 02:57:02.846802 18590 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:34593: Network error: Client connection negotiation failed: client connection to 127.18.20.124:34593: connect: Connection refused (error 111)
I20250111 02:57:02.987506 18590 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } attempt: 4
W20250111 02:57:02.993979 18590 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:34593: Network error: Client connection negotiation failed: client connection to 127.18.20.124:34593: connect: Connection refused (error 111)
I20250111 02:57:03.277442 18590 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } attempt: 5
W20250111 02:57:03.283937 18590 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:34593: Network error: Client connection negotiation failed: client connection to 127.18.20.124:34593: connect: Connection refused (error 111)
I20250111 02:57:03.313017 18665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } attempt: 6
W20250111 02:57:03.323098 18665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:34593: Network error: Client connection negotiation failed: client connection to 127.18.20.124:34593: connect: Connection refused (error 111)
W20250111 02:57:02.450575 18678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:57:02.451728 18680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:02.452474 18667 server_base.cc:1034] running on GCE node
I20250111 02:57:03.672354 18667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:57:03.675206 18667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:57:03.676630 18667 hybrid_clock.cc:648] HybridClock initialized: now 1736564223676596 us; error 46 us; skew 500 ppm
I20250111 02:57:03.677526 18667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:57:03.684108 18667 webserver.cc:458] Webserver started at http://127.18.20.124:34755/ using document root <none> and password file <none>
I20250111 02:57:03.685202 18667 fs_manager.cc:362] Metadata directory not provided
I20250111 02:57:03.685451 18667 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:57:03.685917 18667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:57:03.691232 18667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/data/instance:
uuid: "c34b72b7c70343d3b6f3fae79ce4be6c"
format_stamp: "Formatted at 2025-01-11 02:57:03 on dist-test-slave-09x0"
I20250111 02:57:03.692516 18667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/wal/instance:
uuid: "c34b72b7c70343d3b6f3fae79ce4be6c"
format_stamp: "Formatted at 2025-01-11 02:57:03 on dist-test-slave-09x0"
I20250111 02:57:03.700362 18667 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250111 02:57:03.706543 18692 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:57:03.707803 18667 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250111 02:57:03.708204 18667 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/data,/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/wal
uuid: "c34b72b7c70343d3b6f3fae79ce4be6c"
format_stamp: "Formatted at 2025-01-11 02:57:03 on dist-test-slave-09x0"
I20250111 02:57:03.708577 18667 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:57:03.767302 18667 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:57:03.768889 18667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:57:03.769383 18667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:57:03.836395 18590 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } attempt: 6
I20250111 02:57:03.844878 18667 rpc_server.cc:307] RPC server started. Bound to: 127.18.20.124:34593
I20250111 02:57:03.845049 18745 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.20.124:34593 every 8 connection(s)
I20250111 02:57:03.848351 18667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/data/info.pb
I20250111 02:57:03.858587 18747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:03.864473 18513 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu as pid 18667
I20250111 02:57:03.864965 18513 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-2/wal/instance
I20250111 02:57:03.875619 18747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:57:03.887444 18590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Bootstrap starting.
I20250111 02:57:03.895970 18590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:03.898559 18590 log.cc:826] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Log is configured to *not* fsync() on all Append() calls
I20250111 02:57:03.902865 18747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:57:03.905081 18590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: No bootstrap required, opened a new log
I20250111 02:57:03.917307 18747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:57:03.934394 18590 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:03.935462 18590 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:03.935791 18590 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e336d4ddf044bf5b0b6e4f5af634727, State: Initialized, Role: FOLLOWER
I20250111 02:57:03.936664 18590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:03.942888 18747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c: Bootstrap starting.
I20250111 02:57:03.944789 18758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:03.947022 18758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:03.947593 18758 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:03.949149 18758 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:03.951352 18747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:03.952546 18758 leader_election.cc:290] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8784b9603f8544b79373158aace063af (127.18.20.125:40359), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:03.955431 18747 log.cc:826] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c: Log is configured to *not* fsync() on all Append() calls
I20250111 02:57:03.956657 18590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:57:03.963410 18747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c: No bootstrap required, opened a new log
I20250111 02:57:03.996966 18747 raft_consensus.cc:357] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:03.998081 18747 raft_consensus.cc:383] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:03.998461 18747 raft_consensus.cc:738] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34b72b7c70343d3b6f3fae79ce4be6c, State: Initialized, Role: FOLLOWER
I20250111 02:57:03.999533 18747 consensus_queue.cc:260] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
W20250111 02:57:04.007899 18770 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:57:04.008371 18770 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:57:04.011202 18747 sys_catalog.cc:564] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:57:04.011379 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:04.012451 18719 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e336d4ddf044bf5b0b6e4f5af634727 in term 0.
I20250111 02:57:04.014091 18540 leader_election.cc:304] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c; no voters:
I20250111 02:57:04.015476 18757 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:04.015116 18772 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:04.015925 18757 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:04.016081 18772 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:04.016283 18757 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:04.024587 18698 tablet.cc:2367] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:57:04.029469 18757 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:04.036150 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c"
I20250111 02:57:04.036830 18719 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:03.991957 18522 master_runner.cc:186] Error getting master registration for 127.18.20.125:40359: OK, instance_id { permanent_uuid: "8784b9603f8544b79373158aace063af" instance_seqno: 1736564222001466 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250111 02:57:04.053037 18719 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e336d4ddf044bf5b0b6e4f5af634727 in term 1.
I20250111 02:57:04.053799 18522 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:57:04.055390 18540 leader_election.cc:304] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c; no voters:
I20250111 02:57:04.056387 18758 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:04.067230 18757 leader_election.cc:290] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 1 election: Requested vote from peers 8784b9603f8544b79373158aace063af (127.18.20.125:40359), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:04.079996 18758 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 1 LEADER]: Becoming Leader. State: Replica: 1e336d4ddf044bf5b0b6e4f5af634727, State: Running, Role: LEADER
I20250111 02:57:04.081724 18758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:04.101933 18757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e336d4ddf044bf5b0b6e4f5af634727. Latest consensus state: current_term: 1 leader_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:04.102893 18757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:04.107007 18784 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:57:04.124326 18784 catalog_manager.cc:1485] Initializing Kudu cluster ID...
W20250111 02:57:04.127650 18786 catalog_manager.cc:1559] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:57:04.128146 18786 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:57:04.171378 18544 tablet.cc:2367] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:57:04.129901 18667 master_runner.cc:186] Error getting master registration for 127.18.20.125:40359: OK, instance_id { permanent_uuid: "8784b9603f8544b79373158aace063af" instance_seqno: 1736564222001466 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250111 02:57:04.187397 18667 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:57:04.191602 18719 raft_consensus.cc:1270] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Refusing update from remote peer 1e336d4ddf044bf5b0b6e4f5af634727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:57:04.195533 18758 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:04.236059 18772 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e336d4ddf044bf5b0b6e4f5af634727. Latest consensus state: current_term: 1 leader_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:04.236920 18772 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:04.255209 18788 mvcc.cc:204] Tried to move back new op lower bound from 7112967062215729152 to 7112967061856104448. Current Snapshot: MvccSnapshot[applied={T|T < 7112967062215729152}]
I20250111 02:57:04.269737 18757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:04.270605 18757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:04.271427 18784 catalog_manager.cc:1348] Generated new cluster ID: 505f17df54394056975cd03f4ae103e8
I20250111 02:57:04.271818 18784 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:57:04.273705 18772 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:04.274277 18772 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:04.357367 18784 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:57:04.363266 18784 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:57:04.383598 18665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } attempt: 7
I20250111 02:57:04.414940 18665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Bootstrap starting.
I20250111 02:57:04.422977 18665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:04.441952 18665 log.cc:826] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Log is configured to *not* fsync() on all Append() calls
I20250111 02:57:04.469514 18665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: No bootstrap required, opened a new log
I20250111 02:57:04.470361 18784 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Generated new TSK 0
I20250111 02:57:04.471621 18784 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:57:04.497781 18513 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
/tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.20.65:0
--local_ip_for_outbound_sockets=127.18.20.65
--webserver_interface=127.18.20.65
--webserver_port=0
--tserver_master_addrs=127.18.20.126:45115,127.18.20.125:40359,127.18.20.124:34593
--builtin_ntp_servers=127.18.20.84:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250111 02:57:04.525746 18665 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:04.526985 18665 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:04.527391 18665 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8784b9603f8544b79373158aace063af, State: Initialized, Role: FOLLOWER
I20250111 02:57:04.528612 18665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:04.537248 18797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:04.541375 18797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:04.541567 18638 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:04.542230 18639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af"
I20250111 02:57:04.543458 18640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
W20250111 02:57:04.544484 18640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55964 (request call id 1) took 592 ms (client timeout 575 ms). Trace:
W20250111 02:57:04.544857 18640 rpcz_store.cc:269] 0111 02:57:03.951977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:03.952149 (+ 172us) service_pool.cc:225] Handling call
0111 02:57:04.544400 (+592251us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:04.546618 18639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55964 (request call id 3) took 513 ms (client timeout 554 ms). Trace:
W20250111 02:57:04.546986 18639 rpcz_store.cc:269] 0111 02:57:04.033642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:04.033807 (+ 165us) service_pool.cc:225] Handling call
0111 02:57:04.546553 (+512746us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:04.540987 18665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:57:04.548592 18639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:04.556631 18638 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Refusing update from remote peer 1e336d4ddf044bf5b0b6e4f5af634727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:57:04.558862 18758 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:04.646512 18797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e336d4ddf044bf5b0b6e4f5af634727. Latest consensus state: current_term: 1 leader_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:04.646632 18804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:04.647416 18797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:04.647548 18804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:04.696115 18813 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250111 02:57:04.990612 18758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:04.991459 18758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: LEADER
W20250111 02:57:05.114886 18795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:57:05.115479 18795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:57:05.116086 18795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250111 02:57:05.135092 18786 catalog_manager.cc:1260] Loaded cluster ID: 505f17df54394056975cd03f4ae103e8
I20250111 02:57:05.135442 18786 catalog_manager.cc:1553] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c: loading cluster ID for follower catalog manager: success
I20250111 02:57:05.143601 18786 catalog_manager.cc:1575] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c: acquiring CA information for follower catalog manager: success
I20250111 02:57:05.149657 18786 catalog_manager.cc:1603] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250111 02:57:05.166544 18795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:57:05.168217 18795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.20.65
I20250111 02:57:05.234015 18795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.20.84:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.20.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.20.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.18.20.126:45115,127.18.20.125:40359,127.18.20.124:34593
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.20.65
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4120
TSAN enabled
I20250111 02:57:05.235898 18795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:57:05.238260 18795 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:57:05.297991 18823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:05.410331 18828 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:05.411115 18828 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:05.414757 18565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:05.417603 18828 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:05.418290 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:05.420061 18693 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:05.421417 18828 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:05.494355 18829 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:05.495106 18829 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:05.498534 18565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:05.502053 18829 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:05.501984 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:05.503720 18615 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8784b9603f8544b79373158aace063af; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:05.505376 18829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:05.715517 18813 catalog_manager.cc:1260] Loaded cluster ID: 505f17df54394056975cd03f4ae103e8
I20250111 02:57:05.720710 18813 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: loading cluster ID for follower catalog manager: success
I20250111 02:57:05.733825 18813 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: acquiring CA information for follower catalog manager: success
I20250111 02:57:05.747563 18813 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 02:57:05.959766 18831 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:05.960335 18831 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:05.963395 18565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:05.963806 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:05.965481 18693 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:05.971331 18831 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:05.973150 18832 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:06.114514 18833 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:06.115360 18833 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:06.121644 18565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:06.125265 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:06.127605 18615 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8784b9603f8544b79373158aace063af; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:06.130867 18833 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:06.132040 18833 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:06.543887 18835 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:06.544468 18835 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:06.547530 18835 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:06.547410 18565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:06.548867 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:06.551344 18693 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:06.552176 18835 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250111 02:57:05.307293 18822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:05.307734 18795 server_base.cc:1034] running on GCE node
W20250111 02:57:05.299031 18826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:06.935467 18795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:57:06.944904 18795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:57:06.946501 18795 hybrid_clock.cc:648] HybridClock initialized: now 1736564226946457 us; error 54 us; skew 500 ppm
I20250111 02:57:06.947844 18795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:57:06.975860 18795 webserver.cc:458] Webserver started at http://127.18.20.65:42449/ using document root <none> and password file <none>
I20250111 02:57:06.977421 18795 fs_manager.cc:362] Metadata directory not provided
I20250111 02:57:06.977835 18795 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:57:06.978528 18795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:57:06.986773 18795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/data/instance:
uuid: "8221604733fe4fc0a1fc32de8d6af71c"
format_stamp: "Formatted at 2025-01-11 02:57:06 on dist-test-slave-09x0"
I20250111 02:57:06.988736 18795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/instance:
uuid: "8221604733fe4fc0a1fc32de8d6af71c"
format_stamp: "Formatted at 2025-01-11 02:57:06 on dist-test-slave-09x0"
I20250111 02:57:07.005388 18795 fs_manager.cc:696] Time spent creating directory manager: real 0.016s user 0.009s sys 0.004s
I20250111 02:57:07.020154 18840 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:57:07.021929 18795 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.008s sys 0.000s
I20250111 02:57:07.022498 18795 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal
uuid: "8221604733fe4fc0a1fc32de8d6af71c"
format_stamp: "Formatted at 2025-01-11 02:57:06 on dist-test-slave-09x0"
I20250111 02:57:07.023135 18795 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:57:07.105616 18795 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:57:07.108099 18795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:57:07.108834 18795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:57:07.113580 18845 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:07.114401 18845 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:07.117377 18565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:07.119235 18795 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:57:07.125360 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:07.127619 18693 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:07.129176 18848 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:07.129076 18845 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:07.131621 18795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:57:07.131958 18795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:57:07.132265 18795 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:57:07.132485 18795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:57:07.403882 18795 rpc_server.cc:307] RPC server started. Bound to: 127.18.20.65:34535
I20250111 02:57:07.404021 18954 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.20.65:34535 every 8 connection(s)
I20250111 02:57:07.406600 18795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/data/info.pb
I20250111 02:57:07.415891 18513 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu as pid 18795
I20250111 02:57:07.416515 18513 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/instance
I20250111 02:57:07.432878 18513 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
/tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.20.66:0
--local_ip_for_outbound_sockets=127.18.20.66
--webserver_interface=127.18.20.66
--webserver_port=0
--tserver_master_addrs=127.18.20.126:45115,127.18.20.125:40359,127.18.20.124:34593
--builtin_ntp_servers=127.18.20.84:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250111 02:57:07.507035 18955 heartbeater.cc:346] Connected to a master server at 127.18.20.124:34593
I20250111 02:57:07.507632 18955 heartbeater.cc:463] Registering TS with master...
I20250111 02:57:07.509603 18955 heartbeater.cc:510] Master 127.18.20.124:34593 requested a full tablet report, sending...
I20250111 02:57:07.511149 18960 heartbeater.cc:346] Connected to a master server at 127.18.20.126:45115
I20250111 02:57:07.507025 18956 heartbeater.cc:346] Connected to a master server at 127.18.20.125:40359
I20250111 02:57:07.511523 18960 heartbeater.cc:463] Registering TS with master...
I20250111 02:57:07.511849 18956 heartbeater.cc:463] Registering TS with master...
I20250111 02:57:07.512616 18960 heartbeater.cc:510] Master 127.18.20.126:45115 requested a full tablet report, sending...
I20250111 02:57:07.512738 18956 heartbeater.cc:510] Master 127.18.20.125:40359 requested a full tablet report, sending...
I20250111 02:57:07.515975 18709 ts_manager.cc:194] Registered new tserver with Master: 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:07.516772 18630 ts_manager.cc:194] Registered new tserver with Master: 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:07.518707 18555 ts_manager.cc:194] Registered new tserver with Master: 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:07.521523 18555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.20.65:35345
W20250111 02:57:07.828114 18959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:57:07.828758 18959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:57:07.829536 18959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:57:07.862119 18959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:57:07.863166 18959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.20.66
I20250111 02:57:07.903038 18959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.20.84:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.20.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.20.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.18.20.126:45115,127.18.20.125:40359,127.18.20.124:34593
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.20.66
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4120
TSAN enabled
I20250111 02:57:07.904491 18959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:57:07.906172 18959 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:57:07.928709 18972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:08.025821 18960 heartbeater.cc:502] Master 127.18.20.126:45115 was elected leader, sending a full tablet report...
I20250111 02:57:08.464037 18978 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:08.464548 18978 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:08.466409 18978 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:08.468087 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:08.468015 18565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:08.470606 18616 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8784b9603f8544b79373158aace063af; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:08.471887 18978 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:08.646956 18979 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:08.647457 18979 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:08.648941 18979 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:08.650063 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:08.650478 18565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:08.652029 18695 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:08.653163 18979 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250111 02:57:07.929499 18971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:07.931046 18959 server_base.cc:1034] running on GCE node
W20250111 02:57:07.930205 18974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:09.147572 18959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:57:09.149969 18959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:57:09.151392 18959 hybrid_clock.cc:648] HybridClock initialized: now 1736564229151341 us; error 55 us; skew 500 ppm
I20250111 02:57:09.152253 18959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:57:09.159904 18959 webserver.cc:458] Webserver started at http://127.18.20.66:40865/ using document root <none> and password file <none>
I20250111 02:57:09.160929 18959 fs_manager.cc:362] Metadata directory not provided
I20250111 02:57:09.161173 18959 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:57:09.161648 18959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:57:09.166946 18959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/data/instance:
uuid: "54628fbcec2f4359aa9e26a734d4d29f"
format_stamp: "Formatted at 2025-01-11 02:57:09 on dist-test-slave-09x0"
I20250111 02:57:09.168226 18959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/instance:
uuid: "54628fbcec2f4359aa9e26a734d4d29f"
format_stamp: "Formatted at 2025-01-11 02:57:09 on dist-test-slave-09x0"
I20250111 02:57:09.176060 18959 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250111 02:57:09.182513 18985 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:57:09.183722 18959 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250111 02:57:09.184106 18959 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal
uuid: "54628fbcec2f4359aa9e26a734d4d29f"
format_stamp: "Formatted at 2025-01-11 02:57:09 on dist-test-slave-09x0"
I20250111 02:57:09.184489 18959 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:57:09.251427 18990 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:09.251850 18990 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:09.253584 18990 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:09.254663 18565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:09.255026 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:09.256798 18695 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:09.257357 18959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:57:09.257671 18990 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:09.259263 18959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:57:09.259739 18959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:57:09.262909 18959 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:57:09.267788 18959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:57:09.268038 18959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:57:09.268296 18959 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:57:09.268455 18959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:57:09.434064 18959 rpc_server.cc:307] RPC server started. Bound to: 127.18.20.66:44623
I20250111 02:57:09.434191 19098 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.20.66:44623 every 8 connection(s)
I20250111 02:57:09.436735 18959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/data/info.pb
I20250111 02:57:09.442365 18513 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu as pid 18959
I20250111 02:57:09.443181 18513 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/instance
I20250111 02:57:09.462064 18513 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
/tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.20.67:0
--local_ip_for_outbound_sockets=127.18.20.67
--webserver_interface=127.18.20.67
--webserver_port=0
--tserver_master_addrs=127.18.20.126:45115,127.18.20.125:40359,127.18.20.124:34593
--builtin_ntp_servers=127.18.20.84:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250111 02:57:09.465065 19106 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:09.469527 19106 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:09.473434 19106 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:09.473624 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:09.483498 18565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:09.485860 18616 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8784b9603f8544b79373158aace063af; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:09.487437 19106 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:09.495769 19100 heartbeater.cc:346] Connected to a master server at 127.18.20.125:40359
I20250111 02:57:09.495874 19099 heartbeater.cc:346] Connected to a master server at 127.18.20.124:34593
I20250111 02:57:09.496352 19100 heartbeater.cc:463] Registering TS with master...
I20250111 02:57:09.496533 19099 heartbeater.cc:463] Registering TS with master...
I20250111 02:57:09.497857 19099 heartbeater.cc:510] Master 127.18.20.124:34593 requested a full tablet report, sending...
I20250111 02:57:09.497897 19100 heartbeater.cc:510] Master 127.18.20.125:40359 requested a full tablet report, sending...
I20250111 02:57:09.502689 18630 ts_manager.cc:194] Registered new tserver with Master: 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:09.502722 18709 ts_manager.cc:194] Registered new tserver with Master: 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:09.506789 19101 heartbeater.cc:346] Connected to a master server at 127.18.20.126:45115
I20250111 02:57:09.507221 19101 heartbeater.cc:463] Registering TS with master...
I20250111 02:57:09.508001 19101 heartbeater.cc:510] Master 127.18.20.126:45115 requested a full tablet report, sending...
I20250111 02:57:09.510066 18555 ts_manager.cc:194] Registered new tserver with Master: 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:09.511529 18555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.20.66:60545
W20250111 02:57:09.820117 19110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:57:09.820645 19110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:57:09.821295 19110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:57:09.853497 19110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:57:09.854507 19110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.20.67
I20250111 02:57:09.892642 19110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.20.84:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.20.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.20.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.18.20.126:45115,127.18.20.125:40359,127.18.20.124:34593
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.20.67
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4120
TSAN enabled
I20250111 02:57:09.894081 19110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:57:09.895820 19110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:57:09.916100 19116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:10.015419 19101 heartbeater.cc:502] Master 127.18.20.126:45115 was elected leader, sending a full tablet report...
I20250111 02:57:10.898150 19123 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:10.898779 19123 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:10.901957 19123 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:10.902702 18565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:10.903090 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:10.904827 18616 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8784b9603f8544b79373158aace063af; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:10.905527 19123 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250111 02:57:09.917084 19117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:09.919549 19110 server_base.cc:1034] running on GCE node
W20250111 02:57:09.918485 19119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:11.145428 19110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:57:11.148077 19110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:57:11.149528 19110 hybrid_clock.cc:648] HybridClock initialized: now 1736564231149475 us; error 55 us; skew 500 ppm
I20250111 02:57:11.150542 19110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:57:11.157368 19110 webserver.cc:458] Webserver started at http://127.18.20.67:36419/ using document root <none> and password file <none>
I20250111 02:57:11.158504 19110 fs_manager.cc:362] Metadata directory not provided
I20250111 02:57:11.158757 19110 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:57:11.159286 19110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:57:11.164637 19110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/data/instance:
uuid: "d0cedab82b154d62b78809edc6f28bfa"
format_stamp: "Formatted at 2025-01-11 02:57:11 on dist-test-slave-09x0"
I20250111 02:57:11.165948 19110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/instance:
uuid: "d0cedab82b154d62b78809edc6f28bfa"
format_stamp: "Formatted at 2025-01-11 02:57:11 on dist-test-slave-09x0"
I20250111 02:57:11.168473 19127 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:11.168864 19127 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:11.170333 19127 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:11.171345 18565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:11.171597 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:11.173377 18693 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:11.174306 19127 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:11.175707 19110 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s
I20250111 02:57:11.182390 19129 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:57:11.183840 19110 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20250111 02:57:11.184334 19110 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal
uuid: "d0cedab82b154d62b78809edc6f28bfa"
format_stamp: "Formatted at 2025-01-11 02:57:11 on dist-test-slave-09x0"
I20250111 02:57:11.184890 19110 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:57:11.244138 19110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:57:11.245709 19110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:57:11.246196 19110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:57:11.248947 19110 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:57:11.253814 19110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:57:11.254065 19110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:57:11.254324 19110 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:57:11.254487 19110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:57:11.428236 19110 rpc_server.cc:307] RPC server started. Bound to: 127.18.20.67:35591
I20250111 02:57:11.429044 19241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.20.67:35591 every 8 connection(s)
I20250111 02:57:11.431545 19110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/data/info.pb
I20250111 02:57:11.434907 18513 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu as pid 19110
I20250111 02:57:11.435453 18513 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/instance
I20250111 02:57:11.468696 19247 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:11.474977 19247 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:11.479216 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:11.479216 18565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:11.484671 18615 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8784b9603f8544b79373158aace063af; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:11.485682 19247 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:11.487632 19247 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:11.498634 19242 heartbeater.cc:346] Connected to a master server at 127.18.20.124:34593
I20250111 02:57:11.499147 19244 heartbeater.cc:346] Connected to a master server at 127.18.20.126:45115
I20250111 02:57:11.499248 19242 heartbeater.cc:463] Registering TS with master...
I20250111 02:57:11.499513 19244 heartbeater.cc:463] Registering TS with master...
I20250111 02:57:11.499967 19243 heartbeater.cc:346] Connected to a master server at 127.18.20.125:40359
I20250111 02:57:11.500279 19243 heartbeater.cc:463] Registering TS with master...
I20250111 02:57:11.500828 19244 heartbeater.cc:510] Master 127.18.20.126:45115 requested a full tablet report, sending...
I20250111 02:57:11.500880 19242 heartbeater.cc:510] Master 127.18.20.124:34593 requested a full tablet report, sending...
I20250111 02:57:11.501075 19243 heartbeater.cc:510] Master 127.18.20.125:40359 requested a full tablet report, sending...
I20250111 02:57:11.504508 18555 ts_manager.cc:194] Registered new tserver with Master: d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:11.504933 18709 ts_manager.cc:194] Registered new tserver with Master: d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:11.505355 18630 ts_manager.cc:194] Registered new tserver with Master: d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:11.506384 18555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.20.67:54825
I20250111 02:57:11.532028 18513 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250111 02:57:11.593701 18554 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58712:
name: "default.Table_59d69957b44844d8bb03d66eaa379290"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:11.594538 18553 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58712:
name: "default.Table_abbd7fedf9fa4aa8bdc1fa79bbd09636"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:11.593540 18555 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58712:
name: "default.Table_6254852e82bc4e078fac113005b83d10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:11.597640 18552 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58712:
name: "default.Table_1c9d0b92da364a19a611c6f95b0d66eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:11.600033 18554 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59d69957b44844d8bb03d66eaa379290 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:11.600157 18553 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abbd7fedf9fa4aa8bdc1fa79bbd09636 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:11.601682 18555 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6254852e82bc4e078fac113005b83d10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:11.603325 18552 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c9d0b92da364a19a611c6f95b0d66eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:11.661484 18513 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu with pid 18522
W20250111 02:57:11.688128 18615 connection.cc:537] server connection from 127.0.0.1:55964 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250111 02:57:12.009555 19244 heartbeater.cc:502] Master 127.18.20.126:45115 was elected leader, sending a full tablet report...
W20250111 02:57:12.014856 19244 heartbeater.cc:643] Failed to heartbeat to 127.18.20.126:45115 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111)
W20250111 02:57:12.039695 19101 heartbeater.cc:643] Failed to heartbeat to 127.18.20.126:45115 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111)
W20250111 02:57:12.065671 18960 heartbeater.cc:643] Failed to heartbeat to 127.18.20.126:45115 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111)
I20250111 02:57:12.137224 19288 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:12.137641 19288 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:12.139281 19288 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:12.141155 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
W20250111 02:57:12.141654 18695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111)
I20250111 02:57:12.144308 19291 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:12.144731 19291 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
W20250111 02:57:12.146509 18695 leader_election.cc:336] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115): Network error: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111)
I20250111 02:57:12.147092 18695 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:12.147711 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:12.147900 19288 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:12.148859 19291 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:12.148890 18719 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 1.
I20250111 02:57:12.150278 18615 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c; no voters:
I20250111 02:57:12.151243 19291 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 02:57:12.151652 19291 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e336d4ddf044bf5b0b6e4f5af634727)
I20250111 02:57:12.152129 19291 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 1 FOLLOWER]: Advancing to term 2
W20250111 02:57:12.153084 18616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20250111 02:57:12.158358 18616 leader_election.cc:336] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115): Network error: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111)
I20250111 02:57:12.212028 19291 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:12.215358 19291 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 election: Requested vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:12.216524 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c"
I20250111 02:57:12.217136 18719 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 1 FOLLOWER]: Advancing to term 2
W20250111 02:57:12.221562 18616 leader_election.cc:336] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115): Network error: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111)
I20250111 02:57:12.315853 18719 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 2.
I20250111 02:57:12.317256 18615 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727
I20250111 02:57:12.318305 19291 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:57:12.319897 19291 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 2 LEADER]: Becoming Leader. State: Replica: 8784b9603f8544b79373158aace063af, State: Running, Role: LEADER
I20250111 02:57:12.321362 19291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:12.327574 19294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8784b9603f8544b79373158aace063af. Latest consensus state: current_term: 2 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:12.328250 19294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: LEADER
W20250111 02:57:12.706887 18616 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af -> Peer 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115): Couldn't send request to peer 1e336d4ddf044bf5b0b6e4f5af634727. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:12.937358 18719 raft_consensus.cc:1270] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 2 FOLLOWER]: Refusing update from remote peer 8784b9603f8544b79373158aace063af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250111 02:57:12.938820 19294 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 02:57:12.945025 19300 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8784b9603f8544b79373158aace063af. Latest consensus state: current_term: 2 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:12.945916 19300 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:12.954799 19302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:12.955897 19302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:12.957756 19300 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:12.958426 19300 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:12.978123 19295 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:57:12.987567 19295 catalog_manager.cc:670] Loaded metadata for table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]
I20250111 02:57:12.993835 19295 tablet_loader.cc:96] loaded metadata for tablet 78bbfb4fbb5f44559b444218f2af764c (table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204])
I20250111 02:57:12.995533 19295 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:57:13.000232 19295 catalog_manager.cc:1260] Loaded cluster ID: 505f17df54394056975cd03f4ae103e8
I20250111 02:57:13.000609 19295 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:57:13.006011 19295 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:57:13.010565 19295 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Loaded TSK: 0
I20250111 02:57:13.012039 19295 catalog_manager.cc:1515] Initializing in-progress tserver states...
W20250111 02:57:12.918325 18518 debug-util.cc:398] Leaking SignalData structure 0x7b080002c0c0 after lost signal to thread 19256
W20250111 02:57:13.044517 18518 kernel_stack_watchdog.cc:198] Thread 19256 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 228ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20250111 02:57:13.045749 18513 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
/tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.126:45115
--webserver_interface=127.18.20.126
--webserver_port=36887
--builtin_ntp_servers=127.18.20.84:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.20.126:45115,127.18.20.125:40359,127.18.20.124:34593
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250111 02:57:11.689981 19256 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.20.126:45115)
W20250111 02:57:13.058135 19256 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.18.20.126:45115, user_credentials={real_user=slave}} blocked reactor thread for 1.37032e+06us
W20250111 02:57:13.059475 19256 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.20.126:45115)
I20250111 02:57:13.076834 18629 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_abbd7fedf9fa4aa8bdc1fa79bbd09636"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:13.077311 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_59d69957b44844d8bb03d66eaa379290"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:13.078135 18627 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_6254852e82bc4e078fac113005b83d10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:13.112735 18627 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6254852e82bc4e078fac113005b83d10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:13.112982 18629 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abbd7fedf9fa4aa8bdc1fa79bbd09636 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:13.076064 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_1c9d0b92da364a19a611c6f95b0d66eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:13.114077 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59d69957b44844d8bb03d66eaa379290 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:13.120277 18627 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_39c1d9386c16455fb4c5c359069ca9f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:13.122251 18627 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39c1d9386c16455fb4c5c359069ca9f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:13.125276 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c9d0b92da364a19a611c6f95b0d66eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:13.148061 18626 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_abbd7fedf9fa4aa8bdc1fa79bbd09636" } new_table_name: "default.Table_75ebf81760734c1b902a6a726d386bcb" modify_external_catalogs: true
I20250111 02:57:13.173580 18628 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_59d69957b44844d8bb03d66eaa379290" } new_table_name: "default.Table_7837bb28012f4dd1a27fa1bc12c20356" modify_external_catalogs: true
I20250111 02:57:13.174975 18627 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_39c1d9386c16455fb4c5c359069ca9f2" } modify_external_catalogs: true
I20250111 02:57:13.175603 18627 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_39c1d9386c16455fb4c5c359069ca9f2" } modify_external_catalogs: true
W20250111 02:57:13.191584 18626 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 905624f13bf040c1bed59a0449ad5872 has no consensus state
I20250111 02:57:13.209211 18626 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_1c9d0b92da364a19a611c6f95b0d66eb" } modify_external_catalogs: true
I20250111 02:57:13.209890 18626 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_1c9d0b92da364a19a611c6f95b0d66eb" } modify_external_catalogs: true
W20250111 02:57:13.234966 18628 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a05dc653a5384a50b54e3568798633e8 has no consensus state
W20250111 02:57:13.249135 18616 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 905624f13bf040c1bed59a0449ad5872 (table default.Table_75ebf81760734c1b902a6a726d386bcb [id=cd5815568a674140ac8ed00b7cc8191f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 905624f13bf040c1bed59a0449ad5872 consensus state has no leader
I20250111 02:57:13.256485 19177 tablet_service.cc:1467] Processing CreateTablet for tablet 905624f13bf040c1bed59a0449ad5872 (DEFAULT_TABLE table=default.Table_75ebf81760734c1b902a6a726d386bcb [id=cd5815568a674140ac8ed00b7cc8191f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.258459 19177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 905624f13bf040c1bed59a0449ad5872. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.258998 18627 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet bc9be2d379594fd49354b577147051dd
I20250111 02:57:13.259733 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 78bbfb4fbb5f44559b444218f2af764c (DEFAULT_TABLE table=default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.260932 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78bbfb4fbb5f44559b444218f2af764c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.261595 18890 tablet_service.cc:1467] Processing CreateTablet for tablet 905624f13bf040c1bed59a0449ad5872 (DEFAULT_TABLE table=default.Table_75ebf81760734c1b902a6a726d386bcb [id=cd5815568a674140ac8ed00b7cc8191f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.268958 18626 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 2edada56046f4786aea61d1a5b93c56e
I20250111 02:57:13.263686 18889 tablet_service.cc:1467] Processing CreateTablet for tablet 78bbfb4fbb5f44559b444218f2af764c (DEFAULT_TABLE table=default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.271445 18889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78bbfb4fbb5f44559b444218f2af764c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.276744 18890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 905624f13bf040c1bed59a0449ad5872. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.280542 19034 tablet_service.cc:1467] Processing CreateTablet for tablet 905624f13bf040c1bed59a0449ad5872 (DEFAULT_TABLE table=default.Table_75ebf81760734c1b902a6a726d386bcb [id=cd5815568a674140ac8ed00b7cc8191f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.283680 19034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 905624f13bf040c1bed59a0449ad5872. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.282038 19033 tablet_service.cc:1467] Processing CreateTablet for tablet 78bbfb4fbb5f44559b444218f2af764c (DEFAULT_TABLE table=default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.286434 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78bbfb4fbb5f44559b444218f2af764c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.294757 18888 tablet_service.cc:1467] Processing CreateTablet for tablet a05dc653a5384a50b54e3568798633e8 (DEFAULT_TABLE table=default.Table_7837bb28012f4dd1a27fa1bc12c20356 [id=b2f59358b7c34e0b88ae3d165f2879b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.296095 18888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a05dc653a5384a50b54e3568798633e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.296151 19175 tablet_service.cc:1467] Processing CreateTablet for tablet a05dc653a5384a50b54e3568798633e8 (DEFAULT_TABLE table=default.Table_7837bb28012f4dd1a27fa1bc12c20356 [id=b2f59358b7c34e0b88ae3d165f2879b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.297437 19175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a05dc653a5384a50b54e3568798633e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.300248 19032 tablet_service.cc:1467] Processing CreateTablet for tablet a05dc653a5384a50b54e3568798633e8 (DEFAULT_TABLE table=default.Table_7837bb28012f4dd1a27fa1bc12c20356 [id=b2f59358b7c34e0b88ae3d165f2879b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.301537 19032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a05dc653a5384a50b54e3568798633e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.327041 19321 tablet_bootstrap.cc:492] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:13.331177 18626 master_service.cc:946] ReplaceTablet: received request to replace tablet 905624f13bf040c1bed59a0449ad5872 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:13.343405 19322 tablet_bootstrap.cc:492] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:13.348208 19321 tablet_bootstrap.cc:654] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:13.353806 19322 tablet_bootstrap.cc:654] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:13.353391 18627 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_5876a73b13bf4596ba282611078406f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:13.355234 18627 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5876a73b13bf4596ba282611078406f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:13.356982 19321 log.cc:826] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa: Log is configured to *not* fsync() on all Append() calls
I20250111 02:57:13.362203 19322 log.cc:826] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c: Log is configured to *not* fsync() on all Append() calls
I20250111 02:57:13.369756 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_44c452b417f8496a89679757f4f4846e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:13.372834 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44c452b417f8496a89679757f4f4846e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:13.377751 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_bcc704abf52f4cdf96ca9e8685b2c9e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:13.382318 19321 tablet_bootstrap.cc:492] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:13.382928 19321 ts_tablet_manager.cc:1397] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.012s
W20250111 02:57:13.382248 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcc704abf52f4cdf96ca9e8685b2c9e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:13.389622 18626 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 905624f13bf040c1bed59a0449ad5872
I20250111 02:57:13.390512 19322 tablet_bootstrap.cc:492] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:13.391172 19322 ts_tablet_manager.cc:1397] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.048s user 0.017s sys 0.007s
I20250111 02:57:13.392067 19175 tablet_service.cc:1514] Processing DeleteTablet for tablet 905624f13bf040c1bed59a0449ad5872 with delete_type TABLET_DATA_DELETED (replaced by tablet b7436fff471c4ef08e7b7434d1e12ab4) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:13.395037 19031 tablet_service.cc:1514] Processing DeleteTablet for tablet 905624f13bf040c1bed59a0449ad5872 with delete_type TABLET_DATA_DELETED (replaced by tablet b7436fff471c4ef08e7b7434d1e12ab4) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:13.399628 18625 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_75ebf81760734c1b902a6a726d386bcb" } new_table_name: "default.Table_c74214d35774465eb5516078f8dd33d5" modify_external_catalogs: true
I20250111 02:57:13.401273 18629 master_service.cc:946] ReplaceTablet: received request to replace tablet 905624f13bf040c1bed59a0449ad5872 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:13.409981 18629 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_2be5a9ab588a4c52a74918f049c145c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:13.416898 18629 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2be5a9ab588a4c52a74918f049c145c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:13.417321 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 905624f13bf040c1bed59a0449ad5872 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 905624f13bf040c1bed59a0449ad5872 already in progress: creating tablet
I20250111 02:57:13.423882 18626 catalog_manager.cc:6596] ReplaceTablet: tablet 905624f13bf040c1bed59a0449ad5872 deleted and replaced by tablet b7436fff471c4ef08e7b7434d1e12ab4
W20250111 02:57:13.444063 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 905624f13bf040c1bed59a0449ad5872 because the tablet was not found. No further retry: Not found: Tablet not found: 905624f13bf040c1bed59a0449ad5872
I20250111 02:57:13.447762 18627 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_7837bb28012f4dd1a27fa1bc12c20356" } new_table_name: "default.Table_e2ae83f3846440eba353a2512b8e8f9b" modify_external_catalogs: true
I20250111 02:57:13.454275 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 905624f13bf040c1bed59a0449ad5872 with delete_type TABLET_DATA_DELETED (replaced by tablet b7436fff471c4ef08e7b7434d1e12ab4) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:13.466794 18626 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_5876a73b13bf4596ba282611078406f6" } modify_external_catalogs: true
I20250111 02:57:13.467496 18626 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_5876a73b13bf4596ba282611078406f6" } modify_external_catalogs: true
I20250111 02:57:13.467447 19321 raft_consensus.cc:357] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.468403 19321 raft_consensus.cc:383] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:13.468698 19321 raft_consensus.cc:738] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:13.469743 19321 consensus_queue.cc:260] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.476416 19322 raft_consensus.cc:357] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:13.477413 19322 raft_consensus.cc:383] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:13.477710 19322 raft_consensus.cc:738] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
W20250111 02:57:13.479882 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 905624f13bf040c1bed59a0449ad5872 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 905624f13bf040c1bed59a0449ad5872 already in progress: creating tablet
I20250111 02:57:13.479781 19322 consensus_queue.cc:260] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:13.481722 18625 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b7436fff471c4ef08e7b7434d1e12ab4 has no consensus state
I20250111 02:57:13.491585 18625 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_c74214d35774465eb5516078f8dd33d5" } modify_external_catalogs: true
I20250111 02:57:13.492848 18625 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_c74214d35774465eb5516078f8dd33d5" } modify_external_catalogs: true
I20250111 02:57:13.497453 19321 ts_tablet_manager.cc:1428] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.114s user 0.033s sys 0.012s
I20250111 02:57:13.506629 19321 tablet_bootstrap.cc:492] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:13.513063 19321 tablet_bootstrap.cc:654] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:13.516990 19322 ts_tablet_manager.cc:1428] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.125s user 0.043s sys 0.001s
I20250111 02:57:13.518296 18956 heartbeater.cc:502] Master 127.18.20.125:40359 was elected leader, sending a full tablet report...
I20250111 02:57:13.519366 19322 tablet_bootstrap.cc:492] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:13.525816 19322 tablet_bootstrap.cc:654] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:13.531744 19330 tablet_bootstrap.cc:492] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:13.538314 19100 heartbeater.cc:502] Master 127.18.20.125:40359 was elected leader, sending a full tablet report...
I20250111 02:57:13.549328 19322 tablet_bootstrap.cc:492] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:13.561653 19321 tablet_bootstrap.cc:492] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:13.562362 19321 ts_tablet_manager.cc:1397] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.008s
I20250111 02:57:13.565395 19321 raft_consensus.cc:357] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:13.566555 19322 ts_tablet_manager.cc:1397] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.013s
I20250111 02:57:13.570346 19322 raft_consensus.cc:357] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.573356 19321 raft_consensus.cc:383] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:13.573809 19321 raft_consensus.cc:738] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:13.574184 18623 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet a05dc653a5384a50b54e3568798633e8 (table default.Table_7837bb28012f4dd1a27fa1bc12c20356 [id=b2f59358b7c34e0b88ae3d165f2879b0]). Expected version 1 got 0
I20250111 02:57:13.574703 19321 consensus_queue.cc:260] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:13.577127 19321 ts_tablet_manager.cc:1428] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250111 02:57:13.580826 19322 raft_consensus.cc:383] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:13.581177 19322 raft_consensus.cc:738] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:13.581831 19322 consensus_queue.cc:260] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.584812 19321 tablet_bootstrap.cc:492] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:13.586498 18889 tablet_service.cc:1467] Processing CreateTablet for tablet 38433855438c47318dd579c8fe3bdf7f (DEFAULT_TABLE table=default.Table_5876a73b13bf4596ba282611078406f6 [id=3fb54c7f24824b61ac0aea5abe813383]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.587704 18889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38433855438c47318dd579c8fe3bdf7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.591240 19175 tablet_service.cc:1467] Processing CreateTablet for tablet 38433855438c47318dd579c8fe3bdf7f (DEFAULT_TABLE table=default.Table_5876a73b13bf4596ba282611078406f6 [id=3fb54c7f24824b61ac0aea5abe813383]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.591720 19330 tablet_bootstrap.cc:654] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:13.591862 19243 heartbeater.cc:502] Master 127.18.20.125:40359 was elected leader, sending a full tablet report...
I20250111 02:57:13.592418 19175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38433855438c47318dd579c8fe3bdf7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.594717 19330 log.cc:826] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f: Log is configured to *not* fsync() on all Append() calls
I20250111 02:57:13.595682 18888 tablet_service.cc:1467] Processing CreateTablet for tablet b7436fff471c4ef08e7b7434d1e12ab4 (DEFAULT_TABLE table=default.Table_c74214d35774465eb5516078f8dd33d5 [id=cd5815568a674140ac8ed00b7cc8191f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.596849 18888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7436fff471c4ef08e7b7434d1e12ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.596621 19176 tablet_service.cc:1467] Processing CreateTablet for tablet b7436fff471c4ef08e7b7434d1e12ab4 (DEFAULT_TABLE table=default.Table_c74214d35774465eb5516078f8dd33d5 [id=cd5815568a674140ac8ed00b7cc8191f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.597812 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7436fff471c4ef08e7b7434d1e12ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.598213 19322 ts_tablet_manager.cc:1428] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.031s user 0.005s sys 0.001s
I20250111 02:57:13.599213 19322 tablet_bootstrap.cc:492] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:13.603367 19321 tablet_bootstrap.cc:654] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:13.609664 19322 tablet_bootstrap.cc:654] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:13.615969 18627 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a05dc653a5384a50b54e3568798633e8 consensus state has no leader
I20250111 02:57:13.615965 19034 tablet_service.cc:1467] Processing CreateTablet for tablet 38433855438c47318dd579c8fe3bdf7f (DEFAULT_TABLE table=default.Table_5876a73b13bf4596ba282611078406f6 [id=3fb54c7f24824b61ac0aea5abe813383]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.616344 19321 tablet_bootstrap.cc:492] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:13.616787 19321 ts_tablet_manager.cc:1397] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.005s
I20250111 02:57:13.617137 19034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38433855438c47318dd579c8fe3bdf7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.626297 18627 master_service.cc:946] ReplaceTablet: received request to replace tablet a05dc653a5384a50b54e3568798633e8 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:13.628249 19032 tablet_service.cc:1467] Processing CreateTablet for tablet b7436fff471c4ef08e7b7434d1e12ab4 (DEFAULT_TABLE table=default.Table_c74214d35774465eb5516078f8dd33d5 [id=cd5815568a674140ac8ed00b7cc8191f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.629442 19032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7436fff471c4ef08e7b7434d1e12ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.630688 19322 tablet_bootstrap.cc:492] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:13.631171 19322 ts_tablet_manager.cc:1397] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.001s
I20250111 02:57:13.633491 19328 raft_consensus.cc:491] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:13.633883 19322 raft_consensus.cc:357] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.634114 19328 raft_consensus.cc:513] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.634657 19322 raft_consensus.cc:383] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:13.627574 19321 raft_consensus.cc:357] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.636133 19321 raft_consensus.cc:383] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:13.636435 19321 raft_consensus.cc:738] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:13.638244 19321 consensus_queue.cc:260] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.634945 19322 raft_consensus.cc:738] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:13.645267 19322 consensus_queue.cc:260] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.653862 19322 ts_tablet_manager.cc:1428] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20250111 02:57:13.653671 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_603667a681ce497cbf30adf57beb6f88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:13.655659 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_603667a681ce497cbf30adf57beb6f88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:13.666095 19328 leader_election.cc:290] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:13.669255 18626 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 38433855438c47318dd579c8fe3bdf7f
I20250111 02:57:13.671367 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 38433855438c47318dd579c8fe3bdf7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:51082
W20250111 02:57:13.673185 18617 catalog_manager.cc:4839] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 38433855438c47318dd579c8fe3bdf7f because the tablet was not found. No further retry: Not found: Tablet not found: 38433855438c47318dd579c8fe3bdf7f
I20250111 02:57:13.673949 18629 master_service.cc:946] ReplaceTablet: received request to replace tablet a05dc653a5384a50b54e3568798633e8 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:13.681703 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bbfb4fbb5f44559b444218f2af764c" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:13.682945 18910 raft_consensus.cc:2463] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:13.683303 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet 38433855438c47318dd579c8fe3bdf7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:36190
W20250111 02:57:13.685209 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 38433855438c47318dd579c8fe3bdf7f because the tablet was not found. No further retry: Not found: Tablet not found: 38433855438c47318dd579c8fe3bdf7f
I20250111 02:57:13.688496 18626 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_e2ae83f3846440eba353a2512b8e8f9b" } new_table_name: "default.Table_6883f75ce59a4deaa7ca2c46b1b87f9f" modify_external_catalogs: true
I20250111 02:57:13.693797 19322 tablet_bootstrap.cc:492] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:13.702194 19322 tablet_bootstrap.cc:654] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:13.705781 19330 tablet_bootstrap.cc:492] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:13.706423 19330 ts_tablet_manager.cc:1397] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.175s user 0.024s sys 0.024s
I20250111 02:57:13.707197 19133 leader_election.cc:304] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:13.708454 19328 raft_consensus.cc:2798] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:13.708935 19328 raft_consensus.cc:491] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:13.709380 19328 raft_consensus.cc:3054] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:13.718189 19249 tablet.cc:2367] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:57:13.735468 19322 tablet_bootstrap.cc:492] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:13.735960 19322 ts_tablet_manager.cc:1397] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.042s user 0.014s sys 0.004s
I20250111 02:57:13.740620 19321 ts_tablet_manager.cc:1428] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.124s user 0.017s sys 0.017s
I20250111 02:57:13.741282 19177 tablet_service.cc:1514] Processing DeleteTablet for tablet 38433855438c47318dd579c8fe3bdf7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:13.744868 19321 tablet_bootstrap.cc:492] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:13.746237 19322 raft_consensus.cc:357] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.747016 18627 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet a05dc653a5384a50b54e3568798633e8
I20250111 02:57:13.747107 19322 raft_consensus.cc:383] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:13.747433 19322 raft_consensus.cc:738] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
W20250111 02:57:13.749958 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 38433855438c47318dd579c8fe3bdf7f because the tablet was not found. No further retry: Not found: Tablet not found: 38433855438c47318dd579c8fe3bdf7f
W20250111 02:57:13.747483 18963 tablet.cc:2367] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:57:13.752082 18889 tablet_service.cc:1514] Processing DeleteTablet for tablet a05dc653a5384a50b54e3568798633e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:13.753563 19327 tablet_replica.cc:331] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:13.754446 19327 raft_consensus.cc:2238] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:13.754952 18627 catalog_manager.cc:6596] ReplaceTablet: tablet a05dc653a5384a50b54e3568798633e8 deleted and replaced by tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7
I20250111 02:57:13.748337 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bbfb4fbb5f44559b444218f2af764c" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:13.755035 19327 raft_consensus.cc:2267] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:13.756794 19032 tablet_service.cc:1514] Processing DeleteTablet for tablet a05dc653a5384a50b54e3568798633e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:13.758489 19327 ts_tablet_manager.cc:1905] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:13.764765 19322 consensus_queue.cc:260] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.765772 19175 tablet_service.cc:1514] Processing DeleteTablet for tablet a05dc653a5384a50b54e3568798633e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:13.764415 19328 raft_consensus.cc:513] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:13.767205 19131 leader_election.cc:343] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:13.763859 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet a05dc653a5384a50b54e3568798633e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a05dc653a5384a50b54e3568798633e8 already in progress: creating tablet
I20250111 02:57:13.768498 19328 leader_election.cc:290] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:13.770300 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bbfb4fbb5f44559b444218f2af764c" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
W20250111 02:57:13.771750 19131 leader_election.cc:343] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:13.772315 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bbfb4fbb5f44559b444218f2af764c" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:13.772922 18910 raft_consensus.cc:3054] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:13.777473 19322 ts_tablet_manager.cc:1428] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
I20250111 02:57:13.778476 19322 tablet_bootstrap.cc:492] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:13.766868 19325 tablet_replica.cc:331] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:13.780175 19325 raft_consensus.cc:2238] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:13.780822 19325 raft_consensus.cc:2267] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:13.782289 18627 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_64ebd542bba841a3bd2e93366eef5bd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:13.784161 18627 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64ebd542bba841a3bd2e93366eef5bd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:13.782289 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_65af1591521e4974a28304ce51fe6fb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:13.786118 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65af1591521e4974a28304ce51fe6fb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:13.789076 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_f05e5d6100bc46379e8598ce5d367e03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:13.790863 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f05e5d6100bc46379e8598ce5d367e03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:13.791658 18910 raft_consensus.cc:2463] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:13.793237 19133 leader_election.cc:304] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f
I20250111 02:57:13.794193 19328 raft_consensus.cc:2798] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:13.796728 19322 tablet_bootstrap.cc:654] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:13.794751 19328 raft_consensus.cc:695] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:13.800316 19328 consensus_queue.cc:237] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.801079 19325 ts_tablet_manager.cc:1905] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:13.807123 19327 ts_tablet_manager.cc:1918] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:13.807614 19327 log.cc:1198] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/a05dc653a5384a50b54e3568798633e8
I20250111 02:57:13.808534 19327 ts_tablet_manager.cc:1939] T a05dc653a5384a50b54e3568798633e8 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:13.820760 19321 tablet_bootstrap.cc:654] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:13.824297 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet a05dc653a5384a50b54e3568798633e8 (table default.Table_e2ae83f3846440eba353a2512b8e8f9b [id=b2f59358b7c34e0b88ae3d165f2879b0]) successfully deleted
I20250111 02:57:13.827126 19322 tablet_bootstrap.cc:492] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:13.827620 19322 ts_tablet_manager.cc:1397] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.008s
I20250111 02:57:13.827394 18621 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_bcc704abf52f4cdf96ca9e8685b2c9e3" } new_table_name: "default.Table_552b7806d93a43c9b8b933210c481574" modify_external_catalogs: true
I20250111 02:57:13.830384 19322 raft_consensus.cc:357] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.831161 19322 raft_consensus.cc:383] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:13.831430 19322 raft_consensus.cc:738] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:13.832113 19322 consensus_queue.cc:260] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.834415 19330 raft_consensus.cc:357] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:13.835419 19330 raft_consensus.cc:383] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:13.835745 19330 raft_consensus.cc:738] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:13.838265 19322 ts_tablet_manager.cc:1428] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250111 02:57:13.840036 19330 consensus_queue.cc:260] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:13.850528 19330 ts_tablet_manager.cc:1428] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.144s user 0.026s sys 0.022s
I20250111 02:57:13.853471 19330 tablet_bootstrap.cc:492] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:13.861028 19330 tablet_bootstrap.cc:654] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:13.866899 19329 raft_consensus.cc:491] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:13.867515 19329 raft_consensus.cc:513] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.871047 19329 leader_election.cc:290] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:13.872468 19321 tablet_bootstrap.cc:492] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:13.872915 19321 ts_tablet_manager.cc:1397] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.128s user 0.005s sys 0.007s
I20250111 02:57:13.884330 18628 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_2be5a9ab588a4c52a74918f049c145c6" } modify_external_catalogs: true
I20250111 02:57:13.884970 18628 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_2be5a9ab588a4c52a74918f049c145c6" } modify_external_catalogs: true
I20250111 02:57:13.888434 19330 tablet_bootstrap.cc:492] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:13.888895 19330 ts_tablet_manager.cc:1397] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.004s
I20250111 02:57:13.889003 19321 raft_consensus.cc:357] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.890396 19321 raft_consensus.cc:383] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:13.890690 19321 raft_consensus.cc:738] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:13.891481 19321 consensus_queue.cc:260] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.891897 19330 raft_consensus.cc:357] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.892604 19330 raft_consensus.cc:383] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:13.892869 19330 raft_consensus.cc:738] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:13.893508 19330 consensus_queue.cc:260] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:13.897692 19321 ts_tablet_manager.cc:1428] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.024s user 0.006s sys 0.002s
I20250111 02:57:13.900115 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38433855438c47318dd579c8fe3bdf7f" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:13.901662 19330 ts_tablet_manager.cc:1428] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
W20250111 02:57:13.901952 18842 leader_election.cc:343] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:13.902628 19330 tablet_bootstrap.cc:492] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:13.908994 19321 tablet_bootstrap.cc:492] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:13.922919 19330 tablet_bootstrap.cc:654] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:13.925269 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_91346f650cf44bdb80148802d75e92b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:13.925992 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38433855438c47318dd579c8fe3bdf7f" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:13.927505 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91346f650cf44bdb80148802d75e92b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:13.928100 19325 ts_tablet_manager.cc:1918] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250111 02:57:13.928263 18841 leader_election.cc:343] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:13.928596 19325 log.cc:1198] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/a05dc653a5384a50b54e3568798633e8
I20250111 02:57:13.929034 18841 leader_election.cc:304] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:13.929848 19325 ts_tablet_manager.cc:1939] T a05dc653a5384a50b54e3568798633e8 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:13.930219 19329 raft_consensus.cc:2743] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:13.931488 19321 tablet_bootstrap.cc:654] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:13.933099 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet a05dc653a5384a50b54e3568798633e8 (table default.Table_e2ae83f3846440eba353a2512b8e8f9b [id=b2f59358b7c34e0b88ae3d165f2879b0]) successfully deleted
I20250111 02:57:13.951181 19345 raft_consensus.cc:491] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:13.951812 19345 raft_consensus.cc:513] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:13.949069 19321 tablet_bootstrap.cc:492] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:13.954512 19321 ts_tablet_manager.cc:1397] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.046s user 0.013s sys 0.005s
I20250111 02:57:13.967738 19175 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (DEFAULT_TABLE table=default.Table_bcc704abf52f4cdf96ca9e8685b2c9e3 [id=5d8f4643061740aa96b23a6d4cba8038]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.968971 19175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:13.970506 18626 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 has no consensus state
I20250111 02:57:13.971351 18625 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet b7436fff471c4ef08e7b7434d1e12ab4
I20250111 02:57:13.973898 18889 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (DEFAULT_TABLE table=default.Table_bcc704abf52f4cdf96ca9e8685b2c9e3 [id=5d8f4643061740aa96b23a6d4cba8038]), partition=RANGE (key) PARTITION UNBOUNDED
W20250111 02:57:13.977944 18621 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 consensus state has no leader
I20250111 02:57:13.985117 19032 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (DEFAULT_TABLE table=default.Table_bcc704abf52f4cdf96ca9e8685b2c9e3 [id=5d8f4643061740aa96b23a6d4cba8038]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.986253 19032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:13.987313 19031 tablet_service.cc:1514] Processing DeleteTablet for tablet 905624f13bf040c1bed59a0449ad5872 with delete_type TABLET_DATA_DELETED (replaced by tablet b7436fff471c4ef08e7b7434d1e12ab4) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:13.987830 19034 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfd35c596c34aee9327cb017fecb4a7 (DEFAULT_TABLE table=default.Table_44c452b417f8496a89679757f4f4846e [id=e7a2df18562b4471bcb289ca08bf6574]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.988940 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet b7436fff471c4ef08e7b7434d1e12ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:13.991291 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet b7436fff471c4ef08e7b7434d1e12ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:13.992349 19327 tablet_replica.cc:331] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:13.992487 18888 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfd35c596c34aee9327cb017fecb4a7 (DEFAULT_TABLE table=default.Table_44c452b417f8496a89679757f4f4846e [id=e7a2df18562b4471bcb289ca08bf6574]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:13.993456 19327 raft_consensus.cc:2238] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:13.993716 19326 tablet_replica.cc:331] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:13.993916 18887 tablet_service.cc:1514] Processing DeleteTablet for tablet 905624f13bf040c1bed59a0449ad5872 with delete_type TABLET_DATA_DELETED (replaced by tablet b7436fff471c4ef08e7b7434d1e12ab4) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:13.994112 19327 raft_consensus.cc:2267] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:13.994652 19326 raft_consensus.cc:2238] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:13.995246 19326 raft_consensus.cc:2267] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:13.997097 19327 ts_tablet_manager.cc:1905] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:13.997515 19109 tablet.cc:2367] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:57:13.975023 18889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.002981 18888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bfd35c596c34aee9327cb017fecb4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.006116 19328 raft_consensus.cc:491] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:14.006644 19328 raft_consensus.cc:513] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.008711 19328 leader_election.cc:290] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:14.009855 18625 master_service.cc:946] ReplaceTablet: received request to replace tablet 78bbfb4fbb5f44559b444218f2af764c from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:14.010305 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7436fff471c4ef08e7b7434d1e12ab4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:14.010644 19176 tablet_service.cc:1514] Processing DeleteTablet for tablet b7436fff471c4ef08e7b7434d1e12ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:14.011399 19177 tablet_service.cc:1514] Processing DeleteTablet for tablet 905624f13bf040c1bed59a0449ad5872 with delete_type TABLET_DATA_DELETED (replaced by tablet b7436fff471c4ef08e7b7434d1e12ab4) from {username='slave'} at 127.0.0.1:32868
W20250111 02:57:14.011804 19133 leader_election.cc:343] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: Tablet not RUNNING: STOPPED
I20250111 02:57:14.012113 19325 tablet_replica.cc:331] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:14.012941 19325 raft_consensus.cc:2238] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:14.013445 19325 raft_consensus.cc:2267] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:14.015233 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7436fff471c4ef08e7b7434d1e12ab4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:14.019461 19034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bfd35c596c34aee9327cb017fecb4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.023756 19327 ts_tablet_manager.cc:1918] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:14.024302 19327 log.cc:1198] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/b7436fff471c4ef08e7b7434d1e12ab4
I20250111 02:57:14.025168 19327 ts_tablet_manager.cc:1939] T b7436fff471c4ef08e7b7434d1e12ab4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:14.026820 19327 tablet_replica.cc:331] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:14.027189 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet b7436fff471c4ef08e7b7434d1e12ab4 (table default.Table_c74214d35774465eb5516078f8dd33d5 [id=cd5815568a674140ac8ed00b7cc8191f]) successfully deleted
I20250111 02:57:14.029453 18887 tablet_service.cc:1467] Processing CreateTablet for tablet 894bf33e417244148164fa67cabcc324 (DEFAULT_TABLE table=default.Table_2be5a9ab588a4c52a74918f049c145c6 [id=26177bc1f556458387356abc3170311a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.038502 19033 tablet_service.cc:1467] Processing CreateTablet for tablet 894bf33e417244148164fa67cabcc324 (DEFAULT_TABLE table=default.Table_2be5a9ab588a4c52a74918f049c145c6 [id=26177bc1f556458387356abc3170311a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.039757 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894bf33e417244148164fa67cabcc324. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.052663 19345 leader_election.cc:290] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:14.057031 19326 ts_tablet_manager.cc:1905] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:14.034734 19327 raft_consensus.cc:2238] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:14.063319 19327 raft_consensus.cc:2267] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:14.063830 18887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894bf33e417244148164fa67cabcc324. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.065294 19327 ts_tablet_manager.cc:1905] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:14.072180 18628 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 894bf33e417244148164fa67cabcc324
I20250111 02:57:14.075454 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 894bf33e417244148164fa67cabcc324 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:14.076908 19031 tablet_service.cc:1514] Processing DeleteTablet for tablet 894bf33e417244148164fa67cabcc324 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:14.082672 19327 ts_tablet_manager.cc:1918] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:14.087582 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_a4f3ed4b41d04829b593038860548a13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:14.089416 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4f3ed4b41d04829b593038860548a13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:14.091740 19327 log.cc:1198] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/905624f13bf040c1bed59a0449ad5872
I20250111 02:57:14.092522 19327 ts_tablet_manager.cc:1939] T 905624f13bf040c1bed59a0449ad5872 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:14.094369 19326 ts_tablet_manager.cc:1918] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:14.094835 19326 log.cc:1198] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/905624f13bf040c1bed59a0449ad5872
I20250111 02:57:14.094491 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_c2796c2dda744b6c8c8b1b8a7bed2a19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:14.094429 18622 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_fcfe588eae964e7b9f5af53b5728e639"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:14.096848 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2796c2dda744b6c8c8b1b8a7bed2a19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:14.097234 18622 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcfe588eae964e7b9f5af53b5728e639 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:14.097913 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 905624f13bf040c1bed59a0449ad5872 (table default.Table_c74214d35774465eb5516078f8dd33d5 [id=cd5815568a674140ac8ed00b7cc8191f]) successfully deleted
W20250111 02:57:14.099196 18617 catalog_manager.cc:4839] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 894bf33e417244148164fa67cabcc324 because the tablet was not found. No further retry: Not found: Tablet not found: 894bf33e417244148164fa67cabcc324
I20250111 02:57:14.102115 19326 ts_tablet_manager.cc:1939] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:14.109150 19325 ts_tablet_manager.cc:1905] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:14.113677 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet a05dc653a5384a50b54e3568798633e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:14.116331 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 894bf33e417244148164fa67cabcc324 (DEFAULT_TABLE table=default.Table_2be5a9ab588a4c52a74918f049c145c6 [id=26177bc1f556458387356abc3170311a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.117548 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894bf33e417244148164fa67cabcc324. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.120249 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 905624f13bf040c1bed59a0449ad5872 (table default.Table_c74214d35774465eb5516078f8dd33d5 [id=cd5815568a674140ac8ed00b7cc8191f]) successfully deleted
W20250111 02:57:14.121007 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet b7436fff471c4ef08e7b7434d1e12ab4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7436fff471c4ef08e7b7434d1e12ab4 already in progress: creating tablet
W20250111 02:57:14.121642 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 894bf33e417244148164fa67cabcc324 because the tablet was not found. No further retry: Not found: Tablet not found: 894bf33e417244148164fa67cabcc324
W20250111 02:57:14.119269 19131 leader_election.cc:343] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:14.122128 19131 leader_election.cc:304] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
W20250111 02:57:14.123680 19131 raft_consensus.cc:2681] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250111 02:57:14.124657 19177 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfd35c596c34aee9327cb017fecb4a7 (DEFAULT_TABLE table=default.Table_44c452b417f8496a89679757f4f4846e [id=e7a2df18562b4471bcb289ca08bf6574]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.126628 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet 894bf33e417244148164fa67cabcc324 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:14.141801 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "905624f13bf040c1bed59a0449ad5872" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:14.142545 19197 raft_consensus.cc:2463] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:14.144232 18986 leader_election.cc:304] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
W20250111 02:57:14.145068 18986 raft_consensus.cc:2681] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250111 02:57:14.153646 18625 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 78bbfb4fbb5f44559b444218f2af764c
I20250111 02:57:14.157903 18625 catalog_manager.cc:6596] ReplaceTablet: tablet 78bbfb4fbb5f44559b444218f2af764c deleted and replaced by tablet b715e32011114e6286c0b8e5406258c4
I20250111 02:57:14.158118 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 78bbfb4fbb5f44559b444218f2af764c with delete_type TABLET_DATA_DELETED (replaced by tablet b715e32011114e6286c0b8e5406258c4) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:14.159181 19327 tablet_replica.cc:331] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:14.160210 19327 raft_consensus.cc:2238] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:14.160844 19327 raft_consensus.cc:2267] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:14.161718 19031 tablet_service.cc:1514] Processing DeleteTablet for tablet 78bbfb4fbb5f44559b444218f2af764c with delete_type TABLET_DATA_DELETED (replaced by tablet b715e32011114e6286c0b8e5406258c4) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:14.163175 19326 tablet_replica.cc:331] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:14.163911 19326 raft_consensus.cc:2238] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:14.164438 19326 raft_consensus.cc:2267] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:14.164466 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 38433855438c47318dd579c8fe3bdf7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:14.165441 19327 ts_tablet_manager.cc:1905] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:14.166714 19326 ts_tablet_manager.cc:1905] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:14.183331 19177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bfd35c596c34aee9327cb017fecb4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.194833 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet 78bbfb4fbb5f44559b444218f2af764c with delete_type TABLET_DATA_DELETED (replaced by tablet b715e32011114e6286c0b8e5406258c4) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:14.193931 18630 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_552b7806d93a43c9b8b933210c481574" } new_table_name: "default.Table_b3abde5928cc4ea2b7bcdf9204dcb16d" modify_external_catalogs: true
I20250111 02:57:14.199374 19327 ts_tablet_manager.cc:1918] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:14.199832 18628 master_service.cc:946] ReplaceTablet: received request to replace tablet 78bbfb4fbb5f44559b444218f2af764c from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:14.199867 19327 log.cc:1198] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/78bbfb4fbb5f44559b444218f2af764c
I20250111 02:57:14.204808 19326 ts_tablet_manager.cc:1918] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:14.205327 19326 log.cc:1198] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/78bbfb4fbb5f44559b444218f2af764c
I20250111 02:57:14.206214 19326 ts_tablet_manager.cc:1939] T 78bbfb4fbb5f44559b444218f2af764c P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:14.206810 19327 ts_tablet_manager.cc:1939] T 78bbfb4fbb5f44559b444218f2af764c P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:14.209363 19327 tablet_replica.cc:331] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:14.209249 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 78bbfb4fbb5f44559b444218f2af764c (table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]) successfully deleted
I20250111 02:57:14.209513 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 78bbfb4fbb5f44559b444218f2af764c (table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]) successfully deleted
I20250111 02:57:14.210861 19327 raft_consensus.cc:2238] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:14.211493 19327 raft_consensus.cc:2267] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:14.212828 19321 raft_consensus.cc:357] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.213730 19321 raft_consensus.cc:383] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:14.214083 19321 raft_consensus.cc:738] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:14.214936 19321 consensus_queue.cc:260] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.227314 19327 ts_tablet_manager.cc:1905] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:14.227543 18630 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 consensus state has no leader
I20250111 02:57:14.228114 18625 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_91346f650cf44bdb80148802d75e92b0" } new_table_name: "default.Table_5f45eb0bdbd447c19a7a840a6cd9448d" modify_external_catalogs: true
I20250111 02:57:14.230909 18626 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_64ebd542bba841a3bd2e93366eef5bd3" } modify_external_catalogs: true
I20250111 02:57:14.231531 18626 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_64ebd542bba841a3bd2e93366eef5bd3" } modify_external_catalogs: true
W20250111 02:57:14.242687 18989 leader_election.cc:343] T 905624f13bf040c1bed59a0449ad5872 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 905624f13bf040c1bed59a0449ad5872
I20250111 02:57:14.250236 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet b7436fff471c4ef08e7b7434d1e12ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:14.262506 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_d76c4a77f7d244478357d5a5fc3cdda6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:14.263186 19322 tablet_bootstrap.cc:492] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
W20250111 02:57:14.264451 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d76c4a77f7d244478357d5a5fc3cdda6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:14.267879 18625 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3fa8530e894c498cbbfa73a3ba1c751e has no consensus state
W20250111 02:57:14.269172 18617 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_b3abde5928cc4ea2b7bcdf9204dcb16d [id=5d8f4643061740aa96b23a6d4cba8038], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 consensus state has no leader
I20250111 02:57:14.270996 19322 tablet_bootstrap.cc:654] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:14.281370 19327 ts_tablet_manager.cc:1918] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:14.281956 19327 log.cc:1198] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/38433855438c47318dd579c8fe3bdf7f
I20250111 02:57:14.282886 19327 ts_tablet_manager.cc:1939] T 38433855438c47318dd579c8fe3bdf7f P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:14.285981 19325 ts_tablet_manager.cc:1918] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:14.286170 18622 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_a4f3ed4b41d04829b593038860548a13" } modify_external_catalogs: true
I20250111 02:57:14.286430 19325 log.cc:1198] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/b7436fff471c4ef08e7b7434d1e12ab4
I20250111 02:57:14.287150 19325 ts_tablet_manager.cc:1939] T b7436fff471c4ef08e7b7434d1e12ab4 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:14.288017 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 38433855438c47318dd579c8fe3bdf7f (table default.Table_5876a73b13bf4596ba282611078406f6 [id=3fb54c7f24824b61ac0aea5abe813383]) successfully deleted
W20250111 02:57:14.288781 18617 catalog_manager.cc:4839] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet b7436fff471c4ef08e7b7434d1e12ab4 because the tablet was not found. No further retry: Not found: Tablet not found: b7436fff471c4ef08e7b7434d1e12ab4
I20250111 02:57:14.289081 19321 ts_tablet_manager.cc:1428] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.334s user 0.000s sys 0.006s
I20250111 02:57:14.290122 19321 tablet_bootstrap.cc:492] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:14.286798 18622 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_a4f3ed4b41d04829b593038860548a13" } modify_external_catalogs: true
I20250111 02:57:14.309159 19330 tablet_bootstrap.cc:492] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:14.309644 19330 ts_tablet_manager.cc:1397] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.407s user 0.063s sys 0.084s
I20250111 02:57:14.312621 19330 raft_consensus.cc:357] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.313405 19330 raft_consensus.cc:383] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:14.313692 19330 raft_consensus.cc:738] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:14.314347 19330 consensus_queue.cc:260] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.316608 19330 ts_tablet_manager.cc:1428] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250111 02:57:14.317621 19330 tablet_bootstrap.cc:492] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:14.328228 19322 tablet_bootstrap.cc:492] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:14.328893 19322 ts_tablet_manager.cc:1397] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.005s
I20250111 02:57:14.332188 18622 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet c3bc25cd72414e09aeb5c316d829b801
I20250111 02:57:14.337495 19330 tablet_bootstrap.cc:654] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:14.337718 19034 tablet_service.cc:1514] Processing DeleteTablet for tablet a05dc653a5384a50b54e3568798633e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:14.338485 19326 tablet_replica.cc:331] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:14.338770 19321 tablet_bootstrap.cc:654] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:14.339227 19326 raft_consensus.cc:2238] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:14.339807 19326 raft_consensus.cc:2267] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:14.342006 19326 ts_tablet_manager.cc:1905] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:14.345731 19322 raft_consensus.cc:357] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.346616 19322 raft_consensus.cc:383] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:14.346948 19322 raft_consensus.cc:738] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:14.348248 19322 consensus_queue.cc:260] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.360692 19322 ts_tablet_manager.cc:1428] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.031s user 0.004s sys 0.003s
I20250111 02:57:14.366782 19322 tablet_bootstrap.cc:492] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:14.369138 19330 tablet_bootstrap.cc:492] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:14.369588 19330 ts_tablet_manager.cc:1397] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.052s user 0.016s sys 0.000s
I20250111 02:57:14.372215 19330 raft_consensus.cc:357] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.372954 19330 raft_consensus.cc:383] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:14.373235 19330 raft_consensus.cc:738] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:14.375380 18622 master_service.cc:946] ReplaceTablet: received request to replace tablet 894bf33e417244148164fa67cabcc324 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:14.375090 19330 consensus_queue.cc:260] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.381922 19322 tablet_bootstrap.cc:654] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:14.383118 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_baed23f14d1d4ebb978e45731f534aba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:14.384899 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_baed23f14d1d4ebb978e45731f534aba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:14.392758 18623 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_98577ec96fa84b56955dabfab5411da8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:14.393198 19034 tablet_service.cc:1514] Processing DeleteTablet for tablet 38433855438c47318dd579c8fe3bdf7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:14.393345 19322 tablet_bootstrap.cc:492] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:14.393734 19322 ts_tablet_manager.cc:1397] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.005s
W20250111 02:57:14.394547 18623 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98577ec96fa84b56955dabfab5411da8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:14.396097 19322 raft_consensus.cc:357] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.396791 19322 raft_consensus.cc:383] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:14.397059 19322 raft_consensus.cc:738] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:14.397671 19322 consensus_queue.cc:260] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.398872 19330 ts_tablet_manager.cc:1428] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.029s user 0.008s sys 0.000s
I20250111 02:57:14.406778 19330 tablet_bootstrap.cc:492] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:14.410596 18624 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_b3abde5928cc4ea2b7bcdf9204dcb16d [id=5d8f4643061740aa96b23a6d4cba8038]). Expected version 2 got 0
I20250111 02:57:14.412642 19322 ts_tablet_manager.cc:1428] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20250111 02:57:14.413529 19322 tablet_bootstrap.cc:492] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:14.419046 19322 tablet_bootstrap.cc:654] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:14.420184 19325 tablet_replica.cc:331] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:14.421655 18887 tablet_service.cc:1467] Processing CreateTablet for tablet bfee658632fe46db8f3eb133198993a2 (DEFAULT_TABLE table=default.Table_64ebd542bba841a3bd2e93366eef5bd3 [id=5deae041515148e091db2f4de7580ddf]), partition=RANGE (key) PARTITION UNBOUNDED
W20250111 02:57:14.410984 19309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:57:14.422950 19309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20250111 02:57:14.423029 18887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfee658632fe46db8f3eb133198993a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:14.423764 19309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250111 02:57:14.423918 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet b7436fff471c4ef08e7b7434d1e12ab4 (table default.Table_c74214d35774465eb5516078f8dd33d5 [id=cd5815568a674140ac8ed00b7cc8191f]) successfully deleted
I20250111 02:57:14.429152 18622 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_b3abde5928cc4ea2b7bcdf9204dcb16d [id=5d8f4643061740aa96b23a6d4cba8038]). Expected version 2 got 0
I20250111 02:57:14.436228 19177 tablet_service.cc:1467] Processing CreateTablet for tablet bfee658632fe46db8f3eb133198993a2 (DEFAULT_TABLE table=default.Table_64ebd542bba841a3bd2e93366eef5bd3 [id=5deae041515148e091db2f4de7580ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.437546 19177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfee658632fe46db8f3eb133198993a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.448870 19175 tablet_service.cc:1514] Processing DeleteTablet for tablet 38433855438c47318dd579c8fe3bdf7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:14.435457 19034 tablet_service.cc:1467] Processing CreateTablet for tablet bfee658632fe46db8f3eb133198993a2 (DEFAULT_TABLE table=default.Table_64ebd542bba841a3bd2e93366eef5bd3 [id=5deae041515148e091db2f4de7580ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.456415 19034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfee658632fe46db8f3eb133198993a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.456545 19330 tablet_bootstrap.cc:654] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:14.472476 18622 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_b3abde5928cc4ea2b7bcdf9204dcb16d" } new_table_name: "default.Table_48e1bb18933d43449a0aca4cb1d5e378" modify_external_catalogs: true
I20250111 02:57:14.419031 19326 ts_tablet_manager.cc:1918] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:14.478477 19326 log.cc:1198] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/a05dc653a5384a50b54e3568798633e8
I20250111 02:57:14.479605 19326 ts_tablet_manager.cc:1939] T a05dc653a5384a50b54e3568798633e8 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:14.483902 19325 raft_consensus.cc:2238] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:14.484588 19325 raft_consensus.cc:2267] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:14.486456 19325 ts_tablet_manager.cc:1905] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:14.488811 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet a05dc653a5384a50b54e3568798633e8 (table default.Table_6883f75ce59a4deaa7ca2c46b1b87f9f [id=b2f59358b7c34e0b88ae3d165f2879b0]) successfully deleted
I20250111 02:57:14.494184 18630 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_603667a681ce497cbf30adf57beb6f88" } new_table_name: "default.Table_8ef53496bb764e66aced278fcc3f7ece" modify_external_catalogs: true
I20250111 02:57:14.497249 18626 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet bfee658632fe46db8f3eb133198993a2
I20250111 02:57:14.498517 18621 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_8d238bdc5f53451aab66467655c6fb2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:14.500825 18621 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d238bdc5f53451aab66467655c6fb2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:14.502449 18888 tablet_service.cc:1514] Processing DeleteTablet for tablet bfee658632fe46db8f3eb133198993a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:14.503295 19175 tablet_service.cc:1514] Processing DeleteTablet for tablet bfee658632fe46db8f3eb133198993a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:14.508162 18888 tablet_service.cc:1467] Processing CreateTablet for tablet 1df475e5bf1b4a54afeb0839831e0ac4 (DEFAULT_TABLE table=default.Table_603667a681ce497cbf30adf57beb6f88 [id=581224f40f0c4f03a9224f06d660c883]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.508217 19175 tablet_service.cc:1467] Processing CreateTablet for tablet 1df475e5bf1b4a54afeb0839831e0ac4 (DEFAULT_TABLE table=default.Table_603667a681ce497cbf30adf57beb6f88 [id=581224f40f0c4f03a9224f06d660c883]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.509338 18888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df475e5bf1b4a54afeb0839831e0ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.509348 19175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df475e5bf1b4a54afeb0839831e0ac4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:14.512604 18617 catalog_manager.cc:4839] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet bfee658632fe46db8f3eb133198993a2 because the tablet was not found. No further retry: Not found: Tablet not found: bfee658632fe46db8f3eb133198993a2
I20250111 02:57:14.529700 19032 tablet_service.cc:1514] Processing DeleteTablet for tablet bfee658632fe46db8f3eb133198993a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:14.531004 19033 tablet_service.cc:1467] Processing CreateTablet for tablet 1df475e5bf1b4a54afeb0839831e0ac4 (DEFAULT_TABLE table=default.Table_603667a681ce497cbf30adf57beb6f88 [id=581224f40f0c4f03a9224f06d660c883]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.532135 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df475e5bf1b4a54afeb0839831e0ac4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:14.534173 19309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 02:57:14.534574 19309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:57:14.534910 19309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 02:57:14.540455 19309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 02:57:14.543213 18626 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_edda8de6fbd54f229335e11864afc3e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:14.545253 18626 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edda8de6fbd54f229335e11864afc3e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:14.550489 18624 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_44c452b417f8496a89679757f4f4846e" } modify_external_catalogs: true
I20250111 02:57:14.551105 18624 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_44c452b417f8496a89679757f4f4846e" } modify_external_catalogs: true
W20250111 02:57:14.567219 18622 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 consensus state has no leader
W20250111 02:57:14.569602 18630 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1df475e5bf1b4a54afeb0839831e0ac4 consensus state has no leader
I20250111 02:57:14.577200 19321 tablet_bootstrap.cc:492] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:14.577848 19321 ts_tablet_manager.cc:1397] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.288s user 0.010s sys 0.001s
I20250111 02:57:14.587709 19326 tablet_replica.cc:331] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:14.588858 19326 raft_consensus.cc:2238] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:14.589619 19326 raft_consensus.cc:2267] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:14.592653 19326 ts_tablet_manager.cc:1905] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:14.600759 18888 tablet_service.cc:1467] Processing CreateTablet for tablet 8bfad6c08b4b44c1a664cda3b43acbb1 (DEFAULT_TABLE table=default.Table_65af1591521e4974a28304ce51fe6fb1 [id=7cf2761276494576a4c2be6239374334]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.601949 18888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bfad6c08b4b44c1a664cda3b43acbb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.605248 18889 tablet_service.cc:1467] Processing CreateTablet for tablet 6a2234755d474740a55184660119eb11 (DEFAULT_TABLE table=default.Table_f05e5d6100bc46379e8598ce5d367e03 [id=9493f23c2c874bda94417f245e37e24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.606446 18889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a2234755d474740a55184660119eb11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.612004 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 8bfad6c08b4b44c1a664cda3b43acbb1 (DEFAULT_TABLE table=default.Table_65af1591521e4974a28304ce51fe6fb1 [id=7cf2761276494576a4c2be6239374334]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.613243 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bfad6c08b4b44c1a664cda3b43acbb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.615257 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_2130e8dde62a424788eb31bb9f19c2e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:14.617131 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2130e8dde62a424788eb31bb9f19c2e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:14.642838 19321 raft_consensus.cc:357] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.643890 19321 raft_consensus.cc:383] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:14.644169 19321 raft_consensus.cc:738] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:14.644923 19321 consensus_queue.cc:260] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.644822 18624 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 9bfd35c596c34aee9327cb017fecb4a7
I20250111 02:57:14.647964 19174 tablet_service.cc:1467] Processing CreateTablet for tablet 6a2234755d474740a55184660119eb11 (DEFAULT_TABLE table=default.Table_f05e5d6100bc46379e8598ce5d367e03 [id=9493f23c2c874bda94417f245e37e24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.649123 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a2234755d474740a55184660119eb11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.657946 19173 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfd35c596c34aee9327cb017fecb4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:14.604756 19032 tablet_service.cc:1467] Processing CreateTablet for tablet 8bfad6c08b4b44c1a664cda3b43acbb1 (DEFAULT_TABLE table=default.Table_65af1591521e4974a28304ce51fe6fb1 [id=7cf2761276494576a4c2be6239374334]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.607041 19031 tablet_service.cc:1467] Processing CreateTablet for tablet 6a2234755d474740a55184660119eb11 (DEFAULT_TABLE table=default.Table_f05e5d6100bc46379e8598ce5d367e03 [id=9493f23c2c874bda94417f245e37e24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.675681 19329 raft_consensus.cc:491] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:14.676131 19329 raft_consensus.cc:513] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.678076 19329 leader_election.cc:290] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:14.679158 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd35c596c34aee9327cb017fecb4a7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:14.679383 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfd35c596c34aee9327cb017fecb4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:14.680389 19327 tablet_replica.cc:331] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:14.681180 19327 raft_consensus.cc:2238] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:14.681646 19327 raft_consensus.cc:2267] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:14.675258 19309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.20.84:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.20.126:45115,127.18.20.125:40359,127.18.20.124:34593
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.126:45115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.20.126
--webserver_port=36887
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4120
TSAN enabled
I20250111 02:57:14.684159 19030 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfd35c596c34aee9327cb017fecb4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:14.684772 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd35c596c34aee9327cb017fecb4a7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:14.686224 19309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:57:14.686717 19032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bfad6c08b4b44c1a664cda3b43acbb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.688740 19309 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 02:57:14.688825 19330 tablet_bootstrap.cc:492] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:14.689400 19330 ts_tablet_manager.cc:1397] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.288s user 0.036s sys 0.042s
I20250111 02:57:14.691958 19330 raft_consensus.cc:357] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.692755 19330 raft_consensus.cc:383] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:14.693068 19330 raft_consensus.cc:738] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:14.693871 19330 consensus_queue.cc:260] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.695382 19031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a2234755d474740a55184660119eb11. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:14.699246 18841 leader_election.cc:343] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:14.709234 18624 master_service.cc:946] ReplaceTablet: received request to replace tablet 894bf33e417244148164fa67cabcc324 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:14.710229 19034 tablet_service.cc:1514] Processing DeleteTablet for tablet b7436fff471c4ef08e7b7434d1e12ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:14.716632 18624 master_service.cc:946] ReplaceTablet: received request to replace tablet 1df475e5bf1b4a54afeb0839831e0ac4 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:14.722381 19327 ts_tablet_manager.cc:1905] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:14.713042 18842 leader_election.cc:343] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:14.725726 18842 leader_election.cc:304] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
W20250111 02:57:14.726636 18842 raft_consensus.cc:2681] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250111 02:57:14.728858 18890 tablet_service.cc:1467] Processing CreateTablet for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 (DEFAULT_TABLE table=default.Table_6883f75ce59a4deaa7ca2c46b1b87f9f [id=b2f59358b7c34e0b88ae3d165f2879b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.730378 18890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.735891 19330 ts_tablet_manager.cc:1428] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.046s user 0.005s sys 0.000s
I20250111 02:57:14.742359 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 (DEFAULT_TABLE table=default.Table_6883f75ce59a4deaa7ca2c46b1b87f9f [id=b2f59358b7c34e0b88ae3d165f2879b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.743588 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.745415 19329 raft_consensus.cc:491] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:14.746212 19330 tablet_bootstrap.cc:492] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:14.745901 19329 raft_consensus.cc:513] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.747934 19329 leader_election.cc:290] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:14.749157 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6aa50a4dfe4398a82877b6dfcb5c29" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:14.750629 18842 leader_election.cc:343] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:14.761133 19322 tablet_bootstrap.cc:492] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:14.761600 19322 ts_tablet_manager.cc:1397] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.348s user 0.076s sys 0.056s
I20250111 02:57:14.761512 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6aa50a4dfe4398a82877b6dfcb5c29" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:14.764328 19322 raft_consensus.cc:357] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.765081 19322 raft_consensus.cc:383] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:14.765340 19322 raft_consensus.cc:738] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
W20250111 02:57:14.766357 18841 leader_election.cc:343] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:14.766777 18841 leader_election.cc:304] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:14.767796 19329 raft_consensus.cc:2743] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:14.768632 19133 consensus_peers.cc:487] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78bbfb4fbb5f44559b444218f2af764c. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:14.765959 19322 consensus_queue.cc:260] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.769529 19330 tablet_bootstrap.cc:654] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:14.771181 19322 ts_tablet_manager.cc:1428] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20250111 02:57:14.773295 19322 tablet_bootstrap.cc:492] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:14.778954 19322 tablet_bootstrap.cc:654] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:14.783423 18886 tablet_service.cc:1514] Processing DeleteTablet for tablet 894bf33e417244148164fa67cabcc324 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:14.791828 19321 ts_tablet_manager.cc:1428] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.214s user 0.010s sys 0.000s
I20250111 02:57:14.793211 19322 tablet_bootstrap.cc:492] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:14.798218 18626 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_edda8de6fbd54f229335e11864afc3e3" } modify_external_catalogs: true
I20250111 02:57:14.798846 18626 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_edda8de6fbd54f229335e11864afc3e3" } modify_external_catalogs: true
I20250111 02:57:14.800742 19322 ts_tablet_manager.cc:1397] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.000s
I20250111 02:57:14.803009 19031 tablet_service.cc:1467] Processing CreateTablet for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 (DEFAULT_TABLE table=default.Table_6883f75ce59a4deaa7ca2c46b1b87f9f [id=b2f59358b7c34e0b88ae3d165f2879b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:14.804272 19031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:14.805925 19322 raft_consensus.cc:357] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.806651 19322 raft_consensus.cc:383] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250111 02:57:14.807972 19373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:14.806923 19322 raft_consensus.cc:738] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:14.812420 19322 consensus_queue.cc:260] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.819998 19330 tablet_bootstrap.cc:492] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:14.820524 19330 ts_tablet_manager.cc:1397] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.075s user 0.006s sys 0.008s
I20250111 02:57:14.820401 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_8a622a8a40fa4d97899666729fdb7604"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:14.822259 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a622a8a40fa4d97899666729fdb7604 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:14.824818 19322 ts_tablet_manager.cc:1428] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20250111 02:57:14.825682 19322 tablet_bootstrap.cc:492] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:14.823405 19330 raft_consensus.cc:357] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.824131 18621 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_8ef53496bb764e66aced278fcc3f7ece" } new_table_name: "default.Table_a58de727de864b9a8ecab09048b72fb7" modify_external_catalogs: true
I20250111 02:57:14.831007 19330 raft_consensus.cc:383] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:14.836107 19330 raft_consensus.cc:738] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:14.825559 18622 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_98577ec96fa84b56955dabfab5411da8" } new_table_name: "default.Table_7f4578da9f5e418c9d67a535471b45cd" modify_external_catalogs: true
I20250111 02:57:14.832850 18624 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 1df475e5bf1b4a54afeb0839831e0ac4
I20250111 02:57:14.839831 19330 consensus_queue.cc:260] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.843820 19322 tablet_bootstrap.cc:654] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:14.847288 18888 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df475e5bf1b4a54afeb0839831e0ac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 616655aa75824fffa46f81c4f95d5159) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:14.848064 18624 catalog_manager.cc:6596] ReplaceTablet: tablet 1df475e5bf1b4a54afeb0839831e0ac4 deleted and replaced by tablet 616655aa75824fffa46f81c4f95d5159
I20250111 02:57:14.852852 19327 ts_tablet_manager.cc:1918] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:14.853267 19327 log.cc:1198] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/9bfd35c596c34aee9327cb017fecb4a7
I20250111 02:57:14.853933 19327 ts_tablet_manager.cc:1939] T 9bfd35c596c34aee9327cb017fecb4a7 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:14.855830 19175 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df475e5bf1b4a54afeb0839831e0ac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 616655aa75824fffa46f81c4f95d5159) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:14.860363 19330 ts_tablet_manager.cc:1428] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.040s user 0.003s sys 0.002s
I20250111 02:57:14.861881 19327 tablet_replica.cc:331] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:14.862677 19327 raft_consensus.cc:2238] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:14.867270 19321 tablet_bootstrap.cc:492] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:14.867733 19325 ts_tablet_manager.cc:1918] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:14.868129 19325 log.cc:1198] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/905624f13bf040c1bed59a0449ad5872
I20250111 02:57:14.868840 19325 ts_tablet_manager.cc:1939] T 905624f13bf040c1bed59a0449ad5872 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:14.862056 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 9bfd35c596c34aee9327cb017fecb4a7 (table default.Table_44c452b417f8496a89679757f4f4846e [id=e7a2df18562b4471bcb289ca08bf6574]) successfully deleted
I20250111 02:57:14.875106 19330 tablet_bootstrap.cc:492] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:14.882234 19322 tablet_bootstrap.cc:492] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:14.882700 19322 ts_tablet_manager.cc:1397] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.057s user 0.013s sys 0.000s
I20250111 02:57:14.884245 19327 raft_consensus.cc:2267] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:14.885082 19322 raft_consensus.cc:357] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.885810 19322 raft_consensus.cc:383] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:14.886060 19322 raft_consensus.cc:738] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:14.890700 19330 tablet_bootstrap.cc:654] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:14.891381 19322 consensus_queue.cc:260] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.892632 19326 ts_tablet_manager.cc:1918] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:14.893024 19326 log.cc:1198] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/38433855438c47318dd579c8fe3bdf7f
I20250111 02:57:14.893711 19326 ts_tablet_manager.cc:1939] T 38433855438c47318dd579c8fe3bdf7f P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:14.895659 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 905624f13bf040c1bed59a0449ad5872 (table default.Table_c74214d35774465eb5516078f8dd33d5 [id=cd5815568a674140ac8ed00b7cc8191f]) successfully deleted
I20250111 02:57:14.896740 19322 ts_tablet_manager.cc:1428] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.014s user 0.002s sys 0.005s
I20250111 02:57:14.897575 19321 tablet_bootstrap.cc:654] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:14.897562 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet a05dc653a5384a50b54e3568798633e8 because the tablet was not found. No further retry: Not found: Tablet not found: a05dc653a5384a50b54e3568798633e8
I20250111 02:57:14.898645 19325 tablet_replica.cc:331] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:14.899116 19032 tablet_service.cc:1514] Processing DeleteTablet for tablet 894bf33e417244148164fa67cabcc324 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:13 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:14.899550 19325 raft_consensus.cc:2238] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:14.899890 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_97b1974a358b42c694f22cd86099a2d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:14.901705 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97b1974a358b42c694f22cd86099a2d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:14.902374 19325 raft_consensus.cc:2267] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:14.903111 19327 ts_tablet_manager.cc:1905] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:14.905862 19322 tablet_bootstrap.cc:492] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:14.907660 18623 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_8d238bdc5f53451aab66467655c6fb2b" } modify_external_catalogs: true
I20250111 02:57:14.908278 18623 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_8d238bdc5f53451aab66467655c6fb2b" } modify_external_catalogs: true
I20250111 02:57:14.910072 19325 ts_tablet_manager.cc:1905] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:14.911677 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_39701f385f884aac9233a842fcaf93be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:14.911924 18629 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_7fe71b59485f40a088d2a4f02c8bc6fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:14.913466 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39701f385f884aac9233a842fcaf93be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:14.914239 18629 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fe71b59485f40a088d2a4f02c8bc6fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:14.917220 19031 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df475e5bf1b4a54afeb0839831e0ac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 616655aa75824fffa46f81c4f95d5159) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:14.934115 18627 master_service.cc:946] ReplaceTablet: received request to replace tablet 1df475e5bf1b4a54afeb0839831e0ac4 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:14.935680 18626 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet ec28bba7257a4924892f856bbaef78ec
I20250111 02:57:14.937672 19328 raft_consensus.cc:491] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:14.938175 19328 raft_consensus.cc:513] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.940256 19328 leader_election.cc:290] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:14.943306 19133 leader_election.cc:343] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 38433855438c47318dd579c8fe3bdf7f
I20250111 02:57:14.946393 19330 tablet_bootstrap.cc:492] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:14.946834 19330 ts_tablet_manager.cc:1397] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.008s
I20250111 02:57:14.949198 19330 raft_consensus.cc:357] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.949889 19330 raft_consensus.cc:383] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:14.950315 19330 raft_consensus.cc:738] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:14.950802 19326 tablet_replica.cc:331] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:14.950923 19330 consensus_queue.cc:260] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.951788 19321 tablet_bootstrap.cc:492] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:14.951592 19326 raft_consensus.cc:2238] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:14.952221 19321 ts_tablet_manager.cc:1397] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.085s user 0.010s sys 0.001s
I20250111 02:57:14.954571 19321 raft_consensus.cc:357] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:14.951679 19131 leader_election.cc:343] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: 38433855438c47318dd579c8fe3bdf7f
I20250111 02:57:14.955670 19131 leader_election.cc:304] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:14.956364 19326 raft_consensus.cc:2267] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:14.958155 19328 raft_consensus.cc:2743] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:14.958675 19330 ts_tablet_manager.cc:1428] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250111 02:57:14.955430 19321 raft_consensus.cc:383] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:14.961864 19321 raft_consensus.cc:738] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
W20250111 02:57:14.962940 18622 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5baef8688ede42fca28cfb6b0bdeefa6 has no consensus state
I20250111 02:57:14.962635 19321 consensus_queue.cc:260] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:14.965147 19322 tablet_bootstrap.cc:654] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:14.967531 19330 tablet_bootstrap.cc:492] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:14.972191 19321 ts_tablet_manager.cc:1428] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250111 02:57:14.972929 19326 ts_tablet_manager.cc:1905] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:14.973099 19321 tablet_bootstrap.cc:492] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:14.976585 19330 tablet_bootstrap.cc:654] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:14.981936 18615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.124:34593, user_credentials={real_user=slave}} blocked reactor thread for 46160.8us
I20250111 02:57:14.989522 19326 ts_tablet_manager.cc:1918] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:14.989935 19326 log.cc:1198] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/b7436fff471c4ef08e7b7434d1e12ab4
I20250111 02:57:14.990624 19326 ts_tablet_manager.cc:1939] T b7436fff471c4ef08e7b7434d1e12ab4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:14.993753 19326 tablet_replica.cc:331] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:14.994500 19326 raft_consensus.cc:2238] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:14.994967 19326 raft_consensus.cc:2267] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:14.998440 19326 ts_tablet_manager.cc:1905] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:15.009863 19322 tablet_bootstrap.cc:492] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:15.010301 19322 ts_tablet_manager.cc:1397] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.106s user 0.014s sys 0.000s
W20250111 02:57:15.016979 19131 consensus_peers.cc:487] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa -> Peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Couldn't send request to peer 54628fbcec2f4359aa9e26a734d4d29f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78bbfb4fbb5f44559b444218f2af764c. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:15.023325 19321 tablet_bootstrap.cc:654] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:15.023458 19330 tablet_bootstrap.cc:492] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:15.023864 19330 ts_tablet_manager.cc:1397] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.062s user 0.010s sys 0.004s
I20250111 02:57:15.027169 19330 raft_consensus.cc:357] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.027869 19330 raft_consensus.cc:383] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:15.028126 19330 raft_consensus.cc:738] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:15.030094 19325 ts_tablet_manager.cc:1918] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:15.030498 19325 log.cc:1198] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/894bf33e417244148164fa67cabcc324
I20250111 02:57:15.031157 19325 ts_tablet_manager.cc:1939] T 894bf33e417244148164fa67cabcc324 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:15.012553 19322 raft_consensus.cc:357] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.032282 19322 raft_consensus.cc:383] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:15.032560 19322 raft_consensus.cc:738] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:15.032146 19330 consensus_queue.cc:260] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.039736 19321 tablet_bootstrap.cc:492] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:15.040166 19321 ts_tablet_manager.cc:1397] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.001s
I20250111 02:57:15.042299 19321 raft_consensus.cc:357] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.043258 19330 ts_tablet_manager.cc:1428] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20250111 02:57:15.042838 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 38433855438c47318dd579c8fe3bdf7f (table default.Table_5876a73b13bf4596ba282611078406f6 [id=3fb54c7f24824b61ac0aea5abe813383]) successfully deleted
I20250111 02:57:15.043391 19322 consensus_queue.cc:260] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.049750 19322 ts_tablet_manager.cc:1428] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
W20250111 02:57:15.050208 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet bfee658632fe46db8f3eb133198993a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfee658632fe46db8f3eb133198993a2 already in progress: creating tablet
W20250111 02:57:15.051687 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 9bfd35c596c34aee9327cb017fecb4a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bfd35c596c34aee9327cb017fecb4a7 already in progress: creating tablet
I20250111 02:57:15.053560 19322 tablet_bootstrap.cc:492] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:15.060149 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 894bf33e417244148164fa67cabcc324 (table default.Table_2be5a9ab588a4c52a74918f049c145c6 [id=26177bc1f556458387356abc3170311a]) successfully deleted
I20250111 02:57:15.043000 19321 raft_consensus.cc:383] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:15.062005 19321 raft_consensus.cc:738] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:15.062726 19321 consensus_queue.cc:260] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.063094 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet b7436fff471c4ef08e7b7434d1e12ab4 (table default.Table_c74214d35774465eb5516078f8dd33d5 [id=cd5815568a674140ac8ed00b7cc8191f]) successfully deleted
I20250111 02:57:15.064843 19321 ts_tablet_manager.cc:1428] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20250111 02:57:15.071390 19330 tablet_bootstrap.cc:492] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:15.081712 19330 tablet_bootstrap.cc:654] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:15.082340 19321 tablet_bootstrap.cc:492] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:15.084854 19326 ts_tablet_manager.cc:1918] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:15.085273 19326 log.cc:1198] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/894bf33e417244148164fa67cabcc324
I20250111 02:57:15.085983 19326 ts_tablet_manager.cc:1939] T 894bf33e417244148164fa67cabcc324 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:15.087551 19325 tablet_replica.cc:331] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:15.088330 19325 raft_consensus.cc:2238] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:15.089242 19325 pending_rounds.cc:62] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa: Trying to abort 1 pending ops.
I20250111 02:57:15.089579 19325 pending_rounds.cc:69] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112967101659045888 op_type: NO_OP noop_request { }
I20250111 02:57:15.090660 19325 raft_consensus.cc:2883] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:57:15.091024 19325 raft_consensus.cc:2267] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:15.091318 19322 tablet_bootstrap.cc:654] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:15.093183 19325 ts_tablet_manager.cc:1905] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:15.101310 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 894bf33e417244148164fa67cabcc324 (table default.Table_2be5a9ab588a4c52a74918f049c145c6 [id=26177bc1f556458387356abc3170311a]) successfully deleted
I20250111 02:57:15.102382 19321 tablet_bootstrap.cc:654] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:15.107077 18629 master_service.cc:946] ReplaceTablet: received request to replace tablet bfee658632fe46db8f3eb133198993a2 from {username='slave'} at 127.0.0.1:47522
W20250111 02:57:15.107120 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 1df475e5bf1b4a54afeb0839831e0ac4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1df475e5bf1b4a54afeb0839831e0ac4 already in progress: creating tablet
I20250111 02:57:15.115491 19031 tablet_service.cc:1467] Processing CreateTablet for tablet b5148fa2e6504561bd1b1a19e48993d7 (DEFAULT_TABLE table=default.Table_2130e8dde62a424788eb31bb9f19c2e0 [id=8eeffc36d38543339b1c18fa18b0bac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.115826 18626 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_48e1bb18933d43449a0aca4cb1d5e378 [id=5d8f4643061740aa96b23a6d4cba8038]). Expected version 3 got 0
I20250111 02:57:15.116626 19031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5148fa2e6504561bd1b1a19e48993d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.117892 19322 tablet_bootstrap.cc:492] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:15.118059 18630 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_48e1bb18933d43449a0aca4cb1d5e378 [id=5d8f4643061740aa96b23a6d4cba8038]). Expected version 3 got 0
I20250111 02:57:15.118510 19322 ts_tablet_manager.cc:1397] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.003s
I20250111 02:57:15.120555 19174 tablet_service.cc:1467] Processing CreateTablet for tablet 72bcbe140a82462d909bd6d41bfbd2af (DEFAULT_TABLE table=default.Table_8d238bdc5f53451aab66467655c6fb2b [id=f35e4781910c45ba93c35a24da2b83c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.121659 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72bcbe140a82462d909bd6d41bfbd2af. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:15.123466 18621 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 616655aa75824fffa46f81c4f95d5159 has no consensus state
I20250111 02:57:15.125612 19345 raft_consensus.cc:491] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:15.126076 19345 raft_consensus.cc:513] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.128012 19345 leader_election.cc:290] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:15.121150 19322 raft_consensus.cc:357] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.128741 19322 raft_consensus.cc:383] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:15.129024 19322 raft_consensus.cc:738] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:15.129649 19322 consensus_queue.cc:260] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.135020 19032 tablet_service.cc:1467] Processing CreateTablet for tablet 72bcbe140a82462d909bd6d41bfbd2af (DEFAULT_TABLE table=default.Table_8d238bdc5f53451aab66467655c6fb2b [id=f35e4781910c45ba93c35a24da2b83c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.136231 19032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72bcbe140a82462d909bd6d41bfbd2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.138146 19173 tablet_service.cc:1467] Processing CreateTablet for tablet adf93d013fb84ada9ff86ad964273e00 (DEFAULT_TABLE table=default.Table_baed23f14d1d4ebb978e45731f534aba [id=ec872ce86d1940cb85adae5f3047321d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.139381 19173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf93d013fb84ada9ff86ad964273e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.140375 19175 tablet_service.cc:1467] Processing CreateTablet for tablet 149157c1847a497e88a3136a9c1983e9 (DEFAULT_TABLE table=default.Table_d76c4a77f7d244478357d5a5fc3cdda6 [id=b46d11174b994b818737139b9b2eb13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.141467 19175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 149157c1847a497e88a3136a9c1983e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.144021 19177 tablet_service.cc:1467] Processing CreateTablet for tablet 5baef8688ede42fca28cfb6b0bdeefa6 (DEFAULT_TABLE table=default.Table_7f4578da9f5e418c9d67a535471b45cd [id=154c37dc0dba44bdab5e0315756dc7cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.144652 19330 tablet_bootstrap.cc:492] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:15.145052 19330 ts_tablet_manager.cc:1397] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.000s
I20250111 02:57:15.145490 19177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5baef8688ede42fca28cfb6b0bdeefa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.147573 19330 raft_consensus.cc:357] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.147909 19033 tablet_service.cc:1467] Processing CreateTablet for tablet 149157c1847a497e88a3136a9c1983e9 (DEFAULT_TABLE table=default.Table_d76c4a77f7d244478357d5a5fc3cdda6 [id=b46d11174b994b818737139b9b2eb13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.148320 19330 raft_consensus.cc:383] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:15.148612 19330 raft_consensus.cc:738] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:15.149022 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 149157c1847a497e88a3136a9c1983e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.120451 19176 tablet_service.cc:1467] Processing CreateTablet for tablet b5148fa2e6504561bd1b1a19e48993d7 (DEFAULT_TABLE table=default.Table_2130e8dde62a424788eb31bb9f19c2e0 [id=8eeffc36d38543339b1c18fa18b0bac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.151484 19034 tablet_service.cc:1467] Processing CreateTablet for tablet 5baef8688ede42fca28cfb6b0bdeefa6 (DEFAULT_TABLE table=default.Table_7f4578da9f5e418c9d67a535471b45cd [id=154c37dc0dba44bdab5e0315756dc7cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.152547 19034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5baef8688ede42fca28cfb6b0bdeefa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.149632 18627 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_48e1bb18933d43449a0aca4cb1d5e378 [id=5d8f4643061740aa96b23a6d4cba8038]). Expected version 3 got 0
I20250111 02:57:15.157482 19172 tablet_service.cc:1467] Processing CreateTablet for tablet 3c02166674a140f29e9f655d09532ced (DEFAULT_TABLE table=default.Table_c2796c2dda744b6c8c8b1b8a7bed2a19 [id=820cc3b845124b1ea333e55aa6537fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.159425 19172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c02166674a140f29e9f655d09532ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.161413 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5148fa2e6504561bd1b1a19e48993d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.143944 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd35c596c34aee9327cb017fecb4a7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:15.164451 19327 ts_tablet_manager.cc:1918] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:15.166702 19030 tablet_service.cc:1467] Processing CreateTablet for tablet adf93d013fb84ada9ff86ad964273e00 (DEFAULT_TABLE table=default.Table_baed23f14d1d4ebb978e45731f534aba [id=ec872ce86d1940cb85adae5f3047321d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.167953 19030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf93d013fb84ada9ff86ad964273e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.172791 19330 consensus_queue.cc:260] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.174952 19330 ts_tablet_manager.cc:1428] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.030s user 0.004s sys 0.000s
I20250111 02:57:15.175792 19330 tablet_bootstrap.cc:492] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:15.191095 19027 tablet_service.cc:1467] Processing CreateTablet for tablet f610a4ff1cc34a928f2db73f14834b25 (DEFAULT_TABLE table=default.Table_fcfe588eae964e7b9f5af53b5728e639 [id=f1373922978a4598b888e1863525942e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.191208 19029 tablet_service.cc:1467] Processing CreateTablet for tablet 3c02166674a140f29e9f655d09532ced (DEFAULT_TABLE table=default.Table_c2796c2dda744b6c8c8b1b8a7bed2a19 [id=820cc3b845124b1ea333e55aa6537fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.192606 19027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f610a4ff1cc34a928f2db73f14834b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.194768 19026 tablet_service.cc:1467] Processing CreateTablet for tablet b715e32011114e6286c0b8e5406258c4 (DEFAULT_TABLE table=default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.195336 19170 tablet_service.cc:1467] Processing CreateTablet for tablet f610a4ff1cc34a928f2db73f14834b25 (DEFAULT_TABLE table=default.Table_fcfe588eae964e7b9f5af53b5728e639 [id=f1373922978a4598b888e1863525942e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.196060 19026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b715e32011114e6286c0b8e5406258c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.197243 19170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f610a4ff1cc34a928f2db73f14834b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.178345 19197 raft_consensus.cc:2463] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:15.198731 18627 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_baed23f14d1d4ebb978e45731f534aba" } modify_external_catalogs: true
I20250111 02:57:15.199355 18627 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_baed23f14d1d4ebb978e45731f534aba" } modify_external_catalogs: true
I20250111 02:57:15.203116 18986 leader_election.cc:304] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:15.203466 19029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c02166674a140f29e9f655d09532ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.204075 19345 raft_consensus.cc:2798] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:15.204452 19345 raft_consensus.cc:491] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:15.204770 19345 raft_consensus.cc:3054] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:15.205806 18621 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_48e1bb18933d43449a0aca4cb1d5e378" } new_table_name: "default.Table_0158103117c842e0ae88e8f041658357" modify_external_catalogs: true
I20250111 02:57:15.206386 19171 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa8530e894c498cbbfa73a3ba1c751e (DEFAULT_TABLE table=default.Table_5f45eb0bdbd447c19a7a840a6cd9448d [id=a37b721314134c86b39d1f80b7359f19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.210000 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_acc960ebdd02430ab5aa41a4f75f8578"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:15.211530 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acc960ebdd02430ab5aa41a4f75f8578 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:15.216384 19327 log.cc:1198] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/894bf33e417244148164fa67cabcc324
I20250111 02:57:15.218546 18890 tablet_service.cc:1467] Processing CreateTablet for tablet 72bcbe140a82462d909bd6d41bfbd2af (DEFAULT_TABLE table=default.Table_8d238bdc5f53451aab66467655c6fb2b [id=f35e4781910c45ba93c35a24da2b83c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.219772 18888 tablet_service.cc:1467] Processing CreateTablet for tablet 149157c1847a497e88a3136a9c1983e9 (DEFAULT_TABLE table=default.Table_d76c4a77f7d244478357d5a5fc3cdda6 [id=b46d11174b994b818737139b9b2eb13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.220870 18890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72bcbe140a82462d909bd6d41bfbd2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.223382 18885 tablet_service.cc:1467] Processing CreateTablet for tablet 3c02166674a140f29e9f655d09532ced (DEFAULT_TABLE table=default.Table_c2796c2dda744b6c8c8b1b8a7bed2a19 [id=820cc3b845124b1ea333e55aa6537fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.225327 18884 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa8530e894c498cbbfa73a3ba1c751e (DEFAULT_TABLE table=default.Table_5f45eb0bdbd447c19a7a840a6cd9448d [id=a37b721314134c86b39d1f80b7359f19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.226519 18623 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 72bcbe140a82462d909bd6d41bfbd2af
I20250111 02:57:15.227128 18886 tablet_service.cc:1467] Processing CreateTablet for tablet adf93d013fb84ada9ff86ad964273e00 (DEFAULT_TABLE table=default.Table_baed23f14d1d4ebb978e45731f534aba [id=ec872ce86d1940cb85adae5f3047321d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.228711 18889 tablet_service.cc:1467] Processing CreateTablet for tablet 5baef8688ede42fca28cfb6b0bdeefa6 (DEFAULT_TABLE table=default.Table_7f4578da9f5e418c9d67a535471b45cd [id=154c37dc0dba44bdab5e0315756dc7cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.218550 18887 tablet_service.cc:1467] Processing CreateTablet for tablet b5148fa2e6504561bd1b1a19e48993d7 (DEFAULT_TABLE table=default.Table_2130e8dde62a424788eb31bb9f19c2e0 [id=8eeffc36d38543339b1c18fa18b0bac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.233879 18883 tablet_service.cc:1467] Processing CreateTablet for tablet f610a4ff1cc34a928f2db73f14834b25 (DEFAULT_TABLE table=default.Table_fcfe588eae964e7b9f5af53b5728e639 [id=f1373922978a4598b888e1863525942e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.235128 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet 72bcbe140a82462d909bd6d41bfbd2af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:15.235997 19327 ts_tablet_manager.cc:1939] T 894bf33e417244148164fa67cabcc324 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:15.237315 18882 tablet_service.cc:1467] Processing CreateTablet for tablet b715e32011114e6286c0b8e5406258c4 (DEFAULT_TABLE table=default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.239378 18885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c02166674a140f29e9f655d09532ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.243490 18888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 149157c1847a497e88a3136a9c1983e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.243733 18627 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet adf93d013fb84ada9ff86ad964273e00
I20250111 02:57:15.246948 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet adf93d013fb84ada9ff86ad964273e00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:15.251230 19169 tablet_service.cc:1467] Processing CreateTablet for tablet b715e32011114e6286c0b8e5406258c4 (DEFAULT_TABLE table=default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.255818 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b715e32011114e6286c0b8e5406258c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.209371 19171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa8530e894c498cbbfa73a3ba1c751e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.264891 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 72bcbe140a82462d909bd6d41bfbd2af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:15.270824 19167 tablet_service.cc:1514] Processing DeleteTablet for tablet adf93d013fb84ada9ff86ad964273e00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:15.284520 19321 tablet_bootstrap.cc:492] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:15.284953 19321 ts_tablet_manager.cc:1397] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.203s user 0.018s sys 0.011s
I20250111 02:57:15.285982 18889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5baef8688ede42fca28cfb6b0bdeefa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.286223 19330 tablet_bootstrap.cc:654] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:15.288122 18886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf93d013fb84ada9ff86ad964273e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.293015 18884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa8530e894c498cbbfa73a3ba1c751e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.295861 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b715e32011114e6286c0b8e5406258c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.299080 19025 tablet_service.cc:1514] Processing DeleteTablet for tablet 72bcbe140a82462d909bd6d41bfbd2af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:15.300515 19024 tablet_service.cc:1514] Processing DeleteTablet for tablet adf93d013fb84ada9ff86ad964273e00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:36190
W20250111 02:57:15.305292 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 72bcbe140a82462d909bd6d41bfbd2af because the tablet was not found. No further retry: Not found: Tablet not found: 72bcbe140a82462d909bd6d41bfbd2af
W20250111 02:57:15.281594 18989 leader_election.cc:343] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 9bfd35c596c34aee9327cb017fecb4a7
I20250111 02:57:15.307739 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_064d4893c74f4870bc45572dd481c63c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:15.307868 18627 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_16df6ba33da94ec1a52c4b56c5266344"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:15.309839 18614 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 5baef8688ede42fca28cfb6b0bdeefa6 (table default.Table_7f4578da9f5e418c9d67a535471b45cd [id=154c37dc0dba44bdab5e0315756dc7cf], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5baef8688ede42fca28cfb6b0bdeefa6 consensus state has no leader
W20250111 02:57:15.310213 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet adf93d013fb84ada9ff86ad964273e00 because the tablet was not found. No further retry: Not found: Tablet not found: adf93d013fb84ada9ff86ad964273e00
I20250111 02:57:15.193082 19028 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa8530e894c498cbbfa73a3ba1c751e (DEFAULT_TABLE table=default.Table_5f45eb0bdbd447c19a7a840a6cd9448d [id=a37b721314134c86b39d1f80b7359f19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.317116 19028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa8530e894c498cbbfa73a3ba1c751e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:15.317656 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_064d4893c74f4870bc45572dd481c63c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:15.317961 18626 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_af8c2da70d9a46f7886fe6bedfde4b3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:15.319821 18626 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af8c2da70d9a46f7886fe6bedfde4b3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:15.319684 18627 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16df6ba33da94ec1a52c4b56c5266344 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:15.329394 18887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5148fa2e6504561bd1b1a19e48993d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.339468 18883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f610a4ff1cc34a928f2db73f14834b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.340174 18624 master_service.cc:946] ReplaceTablet: received request to replace tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:15.368978 19322 ts_tablet_manager.cc:1428] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.250s user 0.006s sys 0.001s
I20250111 02:57:15.370296 19322 tablet_bootstrap.cc:492] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:15.373423 18629 master_service.cc:946] ReplaceTablet: received request to replace tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:15.373693 18626 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_7fe71b59485f40a088d2a4f02c8bc6fa" } new_table_name: "default.Table_35d7f864912b40bcb5251a90d40c7849" modify_external_catalogs: true
I20250111 02:57:15.385092 19327 tablet_replica.cc:331] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:15.385205 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 894bf33e417244148164fa67cabcc324 (table default.Table_2be5a9ab588a4c52a74918f049c145c6 [id=26177bc1f556458387356abc3170311a]) successfully deleted
I20250111 02:57:15.385926 19327 raft_consensus.cc:2238] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:15.390102 19325 ts_tablet_manager.cc:1918] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:15.390590 19325 log.cc:1198] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/78bbfb4fbb5f44559b444218f2af764c
I20250111 02:57:15.386554 19327 raft_consensus.cc:2267] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:15.393328 19327 ts_tablet_manager.cc:1905] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:15.350754 19345 raft_consensus.cc:513] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:15.424384 18616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.126:45115, user_credentials={real_user=slave}} blocked reactor thread for 48065us
W20250111 02:57:15.427755 18616 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af -> Peer 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115): Couldn't send request to peer 1e336d4ddf044bf5b0b6e4f5af634727. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250111 02:57:15.445680 18989 leader_election.cc:343] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 9bfd35c596c34aee9327cb017fecb4a7
I20250111 02:57:15.447822 19345 leader_election.cc:290] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:15.448956 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd35c596c34aee9327cb017fecb4a7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:15.449918 19197 raft_consensus.cc:3054] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:15.450304 19322 tablet_bootstrap.cc:654] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:15.452157 18630 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_d76c4a77f7d244478357d5a5fc3cdda6" } modify_external_catalogs: true
I20250111 02:57:15.452813 18630 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_d76c4a77f7d244478357d5a5fc3cdda6" } modify_external_catalogs: true
W20250111 02:57:15.458592 18626 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8e8c72c99ac4c369957b571d74baffe has no consensus state
I20250111 02:57:15.468046 19174 tablet_service.cc:1467] Processing CreateTablet for tablet b8e8c72c99ac4c369957b571d74baffe (DEFAULT_TABLE table=default.Table_35d7f864912b40bcb5251a90d40c7849 [id=772d994bee2d4e34bd48912acecf8d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.469410 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e8c72c99ac4c369957b571d74baffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.475570 18881 tablet_service.cc:1467] Processing CreateTablet for tablet b8e8c72c99ac4c369957b571d74baffe (DEFAULT_TABLE table=default.Table_35d7f864912b40bcb5251a90d40c7849 [id=772d994bee2d4e34bd48912acecf8d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.479008 19173 tablet_service.cc:1467] Processing CreateTablet for tablet 5118d4b9cfbb49e8af6af56180858336 (DEFAULT_TABLE table=default.Table_39701f385f884aac9233a842fcaf93be [id=bb83163ef256406da8650e72eadf7f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.480909 19173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5118d4b9cfbb49e8af6af56180858336. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.481032 18881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e8c72c99ac4c369957b571d74baffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.454957 19325 ts_tablet_manager.cc:1939] T 78bbfb4fbb5f44559b444218f2af764c P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:15.489081 18880 tablet_service.cc:1514] Processing DeleteTablet for tablet bfee658632fe46db8f3eb133198993a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:15.490069 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 15) took 1296 ms. Trace:
I20250111 02:57:15.492497 19167 tablet_service.cc:1467] Processing CreateTablet for tablet ede92343d9b54c29b8173a5eafcf68c6 (DEFAULT_TABLE table=default.Table_97b1974a358b42c694f22cd86099a2d6 [id=3c56915488654cc39eb7aade5982cfe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.493791 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ede92343d9b54c29b8173a5eafcf68c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.465724 18628 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_48e1bb18933d43449a0aca4cb1d5e378 [id=5d8f4643061740aa96b23a6d4cba8038]). Expected version 3 got 0
W20250111 02:57:15.497064 18621 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 consensus state has no leader
I20250111 02:57:15.497901 18880 tablet_service.cc:1467] Processing CreateTablet for tablet ede92343d9b54c29b8173a5eafcf68c6 (DEFAULT_TABLE table=default.Table_97b1974a358b42c694f22cd86099a2d6 [id=3c56915488654cc39eb7aade5982cfe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.500553 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ede92343d9b54c29b8173a5eafcf68c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.502952 18630 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 149157c1847a497e88a3136a9c1983e9
I20250111 02:57:15.506353 19321 raft_consensus.cc:357] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.507340 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 149157c1847a497e88a3136a9c1983e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:15.508085 19321 raft_consensus.cc:383] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:15.508460 19321 raft_consensus.cc:738] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:15.508530 18623 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_8b6452ba06af45bab320b50842e527d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:15.509367 19321 consensus_queue.cc:260] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:15.510279 18623 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b6452ba06af45bab320b50842e527d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:15.510510 18627 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_16df6ba33da94ec1a52c4b56c5266344" } modify_external_catalogs: true
I20250111 02:57:15.511231 18627 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_16df6ba33da94ec1a52c4b56c5266344" } modify_external_catalogs: true
I20250111 02:57:15.513556 18879 tablet_service.cc:1514] Processing DeleteTablet for tablet 149157c1847a497e88a3136a9c1983e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:15.517587 18624 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7
I20250111 02:57:15.521607 18879 tablet_service.cc:1514] Processing DeleteTablet for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 with delete_type TABLET_DATA_DELETED (replaced by tablet f1802e7349934fbb9d5f7eda37c3df20) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:15.521937 18624 catalog_manager.cc:6596] ReplaceTablet: tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 deleted and replaced by tablet f1802e7349934fbb9d5f7eda37c3df20
I20250111 02:57:15.524731 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 616655aa75824fffa46f81c4f95d5159 (DEFAULT_TABLE table=default.Table_a58de727de864b9a8ecab09048b72fb7 [id=581224f40f0c4f03a9224f06d660c883]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.526986 18878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 616655aa75824fffa46f81c4f95d5159. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.532383 18879 tablet_service.cc:1467] Processing CreateTablet for tablet 066c87476a854e169fd4f5f1cdbb49f7 (DEFAULT_TABLE table=default.Table_8a622a8a40fa4d97899666729fdb7604 [id=6c02a835750c4952a48c4089e4fe120c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.534659 18879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 066c87476a854e169fd4f5f1cdbb49f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.515918 19168 tablet_service.cc:1467] Processing CreateTablet for tablet 616655aa75824fffa46f81c4f95d5159 (DEFAULT_TABLE table=default.Table_a58de727de864b9a8ecab09048b72fb7 [id=581224f40f0c4f03a9224f06d660c883]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.554282 19168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 616655aa75824fffa46f81c4f95d5159. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.523460 19166 tablet_service.cc:1514] Processing DeleteTablet for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 with delete_type TABLET_DATA_DELETED (replaced by tablet f1802e7349934fbb9d5f7eda37c3df20) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:15.525046 19165 tablet_service.cc:1467] Processing CreateTablet for tablet 066c87476a854e169fd4f5f1cdbb49f7 (DEFAULT_TABLE table=default.Table_8a622a8a40fa4d97899666729fdb7604 [id=6c02a835750c4952a48c4089e4fe120c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.558996 19165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 066c87476a854e169fd4f5f1cdbb49f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.567346 18882 tablet_service.cc:1467] Processing CreateTablet for tablet 5118d4b9cfbb49e8af6af56180858336 (DEFAULT_TABLE table=default.Table_39701f385f884aac9233a842fcaf93be [id=bb83163ef256406da8650e72eadf7f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.568560 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5118d4b9cfbb49e8af6af56180858336. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.572074 18623 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_0aaf5fe148c54a5fbd6e0dc1c58e0e72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:15.573976 18623 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0aaf5fe148c54a5fbd6e0dc1c58e0e72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:15.525532 19197 raft_consensus.cc:2463] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:15.584237 18627 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 71bbfa7040f643599a22d64e733514df
I20250111 02:57:15.587884 18627 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_7bf5032295a34c379d00c2d9ad7e417f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:15.592505 18627 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7bf5032295a34c379d00c2d9ad7e417f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:15.490526 19325 rpcz_store.cc:276] 0111 02:57:14.194111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:14.194260 (+ 149us) service_pool.cc:225] Handling call
0111 02:57:15.075783 (+881523us) ts_tablet_manager.cc:1145] Deleting tablet 78bbfb4fbb5f44559b444218f2af764c
0111 02:57:15.087411 (+ 11628us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:15.142934 (+ 55523us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:15.389831 (+246897us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:15.490001 (+100170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":880481}
I20250111 02:57:15.609298 19345 raft_consensus.cc:491] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:15.610004 19345 raft_consensus.cc:513] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.613806 19031 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfd35c596c34aee9327cb017fecb4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:15.613948 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6aa50a4dfe4398a82877b6dfcb5c29" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:15.614871 18910 raft_consensus.cc:2463] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:15.615011 19326 tablet_replica.cc:331] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:15.616643 19034 tablet_service.cc:1467] Processing CreateTablet for tablet ede92343d9b54c29b8173a5eafcf68c6 (DEFAULT_TABLE table=default.Table_97b1974a358b42c694f22cd86099a2d6 [id=3c56915488654cc39eb7aade5982cfe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.618418 19034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ede92343d9b54c29b8173a5eafcf68c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.623370 19322 tablet_bootstrap.cc:492] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:15.623811 19322 ts_tablet_manager.cc:1397] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.254s user 0.006s sys 0.009s
I20250111 02:57:15.621726 18986 leader_election.cc:304] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:15.629403 18627 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_1c2c853d9f8446d4aa94f39f9eeb1d9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:15.632540 18627 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c2c853d9f8446d4aa94f39f9eeb1d9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:15.648223 19026 tablet_service.cc:1467] Processing CreateTablet for tablet 5118d4b9cfbb49e8af6af56180858336 (DEFAULT_TABLE table=default.Table_39701f385f884aac9233a842fcaf93be [id=bb83163ef256406da8650e72eadf7f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.649369 19322 raft_consensus.cc:357] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.650205 19322 raft_consensus.cc:383] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:15.650485 19322 raft_consensus.cc:738] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:15.650482 19030 tablet_service.cc:1467] Processing CreateTablet for tablet b8e8c72c99ac4c369957b571d74baffe (DEFAULT_TABLE table=default.Table_35d7f864912b40bcb5251a90d40c7849 [id=772d994bee2d4e34bd48912acecf8d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.658080 19329 raft_consensus.cc:491] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:15.658576 19329 raft_consensus.cc:513] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.660662 19329 leader_election.cc:290] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:15.660634 19031 tablet_service.cc:1514] Processing DeleteTablet for tablet 149157c1847a497e88a3136a9c1983e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:15.663290 19326 raft_consensus.cc:2238] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:15.664098 19326 raft_consensus.cc:2267] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:15.667668 19345 leader_election.cc:290] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:15.669569 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfee658632fe46db8f3eb133198993a2" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:15.671214 18841 leader_election.cc:343] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:15.672106 19032 tablet_service.cc:1514] Processing DeleteTablet for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 with delete_type TABLET_DATA_DELETED (replaced by tablet f1802e7349934fbb9d5f7eda37c3df20) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:15.672211 19322 consensus_queue.cc:260] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.673776 19031 tablet_service.cc:1467] Processing CreateTablet for tablet 616655aa75824fffa46f81c4f95d5159 (DEFAULT_TABLE table=default.Table_a58de727de864b9a8ecab09048b72fb7 [id=581224f40f0c4f03a9224f06d660c883]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.675657 19032 tablet_service.cc:1467] Processing CreateTablet for tablet 066c87476a854e169fd4f5f1cdbb49f7 (DEFAULT_TABLE table=default.Table_8a622a8a40fa4d97899666729fdb7604 [id=6c02a835750c4952a48c4089e4fe120c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.676574 18989 leader_election.cc:304] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:15.678341 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfee658632fe46db8f3eb133198993a2" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:15.678822 19322 ts_tablet_manager.cc:1428] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.055s user 0.006s sys 0.000s
I20250111 02:57:15.679487 19054 raft_consensus.cc:2463] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:15.681330 18842 leader_election.cc:304] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters: d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:15.684139 19329 raft_consensus.cc:2798] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:15.684495 19329 raft_consensus.cc:491] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:15.684839 19329 raft_consensus.cc:3054] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:15.687386 19322 tablet_bootstrap.cc:492] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:15.687631 19345 raft_consensus.cc:2798] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:15.688035 19345 raft_consensus.cc:491] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:15.688366 19345 raft_consensus.cc:3054] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:15.690495 19327 ts_tablet_manager.cc:1918] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:15.704644 18627 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_0aaf5fe148c54a5fbd6e0dc1c58e0e72" } new_table_name: "default.Table_154c30276c8d4a8e805dc2fb85867bbf" modify_external_catalogs: true
I20250111 02:57:15.710135 19395 raft_consensus.cc:491] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:15.710642 19395 raft_consensus.cc:513] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.713163 18624 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_a58de727de864b9a8ecab09048b72fb7" } modify_external_catalogs: true
I20250111 02:57:15.713794 18624 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_a58de727de864b9a8ecab09048b72fb7" } modify_external_catalogs: true
I20250111 02:57:15.716213 18623 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_af8c2da70d9a46f7886fe6bedfde4b3a" } new_table_name: "default.Table_292c103f235f421db8dacf834cca2c53" modify_external_catalogs: true
I20250111 02:57:15.718546 19322 tablet_bootstrap.cc:654] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:15.718472 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfad6c08b4b44c1a664cda3b43acbb1" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:15.690896 19327 log.cc:1198] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/1df475e5bf1b4a54afeb0839831e0ac4
I20250111 02:57:15.719599 19327 ts_tablet_manager.cc:1939] T 1df475e5bf1b4a54afeb0839831e0ac4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
W20250111 02:57:15.720232 18842 leader_election.cc:343] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:15.721382 19326 ts_tablet_manager.cc:1905] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:15.722115 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 1df475e5bf1b4a54afeb0839831e0ac4 (table default.Table_a58de727de864b9a8ecab09048b72fb7 [id=581224f40f0c4f03a9224f06d660c883]) successfully deleted
I20250111 02:57:15.722841 19327 tablet_replica.cc:331] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:15.731848 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfad6c08b4b44c1a664cda3b43acbb1" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:15.732826 19197 raft_consensus.cc:2463] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
W20250111 02:57:15.735803 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 72bcbe140a82462d909bd6d41bfbd2af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72bcbe140a82462d909bd6d41bfbd2af already in progress: creating tablet
W20250111 02:57:15.736615 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet adf93d013fb84ada9ff86ad964273e00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adf93d013fb84ada9ff86ad964273e00 already in progress: creating tablet
I20250111 02:57:15.743719 18841 leader_election.cc:304] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f
I20250111 02:57:15.748515 19330 tablet_bootstrap.cc:492] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:15.749074 19330 ts_tablet_manager.cc:1397] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.573s user 0.035s sys 0.003s
I20250111 02:57:15.753106 19330 raft_consensus.cc:357] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.753983 19330 raft_consensus.cc:383] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:15.754261 19330 raft_consensus.cc:738] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:15.754971 19330 consensus_queue.cc:260] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.757171 19330 ts_tablet_manager.cc:1428] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250111 02:57:15.758549 18629 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_a4c544f93b2a4ad5afd4aa78f3989b6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:15.760397 18629 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4c544f93b2a4ad5afd4aa78f3989b6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:15.729357 19395 leader_election.cc:290] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:15.764869 19395 raft_consensus.cc:2798] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250111 02:57:15.768155 18627 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0dbb63ab78f84995ae6d9fb2aae78dcf has no consensus state
I20250111 02:57:15.772564 19395 raft_consensus.cc:491] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:15.773108 19395 raft_consensus.cc:3054] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:15.757926 19026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5118d4b9cfbb49e8af6af56180858336. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.796093 19030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e8c72c99ac4c369957b571d74baffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.799397 19032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 066c87476a854e169fd4f5f1cdbb49f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.787763 19329 raft_consensus.cc:513] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.802714 19327 raft_consensus.cc:2238] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:15.805457 19327 raft_consensus.cc:2267] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:15.806427 19031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 616655aa75824fffa46f81c4f95d5159. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.816449 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfee658632fe46db8f3eb133198993a2" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:15.818362 19054 raft_consensus.cc:3054] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:15.819612 19033 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe4e60db4244a9a9696153b630c575b (DEFAULT_TABLE table=default.Table_064d4893c74f4870bc45572dd481c63c [id=b0a0ff2f4d554503a18d05e0ff51402d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.820863 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe4e60db4244a9a9696153b630c575b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.825538 19330 tablet_bootstrap.cc:492] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:15.826390 19398 raft_consensus.cc:491] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:15.826903 19398 raft_consensus.cc:513] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.834329 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_f731216a607f4fdaadd2e8dbedf7552d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:15.836009 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f731216a607f4fdaadd2e8dbedf7552d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:15.594933 19325 tablet_replica.cc:331] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:15.842204 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_0d373957ef8548a88268ff1eaf56f6a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:15.843986 18628 master_service.cc:946] ReplaceTablet: received request to replace tablet ede92343d9b54c29b8173a5eafcf68c6 from {username='slave'} at 127.0.0.1:47522
W20250111 02:57:15.847326 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d373957ef8548a88268ff1eaf56f6a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:15.850060 19399 raft_consensus.cc:491] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:15.850538 18626 master_service.cc:946] ReplaceTablet: received request to replace tablet 3fa8530e894c498cbbfa73a3ba1c751e from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:15.850808 19399 raft_consensus.cc:513] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.850438 19329 leader_election.cc:290] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:15.853819 18627 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_acc960ebdd02430ab5aa41a4f75f8578" } modify_external_catalogs: true
I20250111 02:57:15.854528 18627 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_acc960ebdd02430ab5aa41a4f75f8578" } modify_external_catalogs: true
I20250111 02:57:15.855974 19327 ts_tablet_manager.cc:1905] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:15.854578 19325 raft_consensus.cc:2238] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:15.856575 19325 raft_consensus.cc:2267] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus is shut down!
W20250111 02:57:15.855422 18841 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 111808us
I20250111 02:57:15.858151 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfee658632fe46db8f3eb133198993a2" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:15.858479 19325 ts_tablet_manager.cc:1905] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:15.859687 18621 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_28852db6fe0d4d5bae2db5ad7fa3ac63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:15.860826 18623 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b54311f4269d4575b8998d7f545000e7 consensus state has no leader
W20250111 02:57:15.861486 18621 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28852db6fe0d4d5bae2db5ad7fa3ac63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:15.861286 18841 leader_election.cc:343] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:15.864265 18879 tablet_service.cc:1467] Processing CreateTablet for tablet d8128b3941d54ce2a72bf2ae762a774f (DEFAULT_TABLE table=default.Table_8b6452ba06af45bab320b50842e527d8 [id=0fa5e1f97e984916a703c7b7da0f1087]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.865418 18879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8128b3941d54ce2a72bf2ae762a774f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.865104 19395 raft_consensus.cc:513] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.869731 18882 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe4e60db4244a9a9696153b630c575b (DEFAULT_TABLE table=default.Table_064d4893c74f4870bc45572dd481c63c [id=b0a0ff2f4d554503a18d05e0ff51402d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.870923 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe4e60db4244a9a9696153b630c575b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.881424 19321 ts_tablet_manager.cc:1428] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.596s user 0.007s sys 0.003s
W20250111 02:57:15.881774 19321 ts_tablet_manager.cc:1460] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa: Tablet startup took 596ms
I20250111 02:57:15.881912 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfad6c08b4b44c1a664cda3b43acbb1" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:15.882541 19197 raft_consensus.cc:3054] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:15.885615 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 78bbfb4fbb5f44559b444218f2af764c (table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]) successfully deleted
I20250111 02:57:15.888010 19392 raft_consensus.cc:491] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:15.888867 19392 raft_consensus.cc:513] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.891404 19197 raft_consensus.cc:2463] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:15.892617 18841 leader_election.cc:304] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:15.893415 19329 raft_consensus.cc:2798] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:15.898608 19398 leader_election.cc:290] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:15.902277 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfad6c08b4b44c1a664cda3b43acbb1" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:15.903259 19399 leader_election.cc:290] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:15.904999 18842 leader_election.cc:343] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:15.905197 18624 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 616655aa75824fffa46f81c4f95d5159
I20250111 02:57:15.915122 19400 raft_consensus.cc:491] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:15.915292 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet 616655aa75824fffa46f81c4f95d5159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:15.916322 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4feb96efdfc44fe4a7cd46dc8deb4fe7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:15.916350 19400 raft_consensus.cc:513] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.916721 18880 tablet_service.cc:1467] Processing CreateTablet for tablet 29561e984e6d40fea032075f0531dd1f (DEFAULT_TABLE table=default.Table_acc960ebdd02430ab5aa41a4f75f8578 [id=dc8029d5413f4e838d330383cf0b6f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.917928 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29561e984e6d40fea032075f0531dd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.919715 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2234755d474740a55184660119eb11" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:15.924468 19024 tablet_service.cc:1467] Processing CreateTablet for tablet 29561e984e6d40fea032075f0531dd1f (DEFAULT_TABLE table=default.Table_acc960ebdd02430ab5aa41a4f75f8578 [id=dc8029d5413f4e838d330383cf0b6f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.923380 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2234755d474740a55184660119eb11" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:15.880602 19395 leader_election.cc:290] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:15.926163 18841 leader_election.cc:343] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:15.929790 18841 leader_election.cc:343] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:15.929909 19024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29561e984e6d40fea032075f0531dd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.930367 18878 tablet_service.cc:1467] Processing CreateTablet for tablet b54311f4269d4575b8998d7f545000e7 (DEFAULT_TABLE table=default.Table_292c103f235f421db8dacf834cca2c53 [id=b091f6cd63104693abbdd739bca159c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.933318 18878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b54311f4269d4575b8998d7f545000e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.935864 18886 tablet_service.cc:1467] Processing CreateTablet for tablet f1802e7349934fbb9d5f7eda37c3df20 (DEFAULT_TABLE table=default.Table_6883f75ce59a4deaa7ca2c46b1b87f9f [id=b2f59358b7c34e0b88ae3d165f2879b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.937108 18886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1802e7349934fbb9d5f7eda37c3df20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:15.938128 19329 raft_consensus.cc:695] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 LEADER]: Becoming Leader. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Running, Role: LEADER
I20250111 02:57:15.937237 19025 tablet_service.cc:1467] Processing CreateTablet for tablet b54311f4269d4575b8998d7f545000e7 (DEFAULT_TABLE table=default.Table_292c103f235f421db8dacf834cca2c53 [id=b091f6cd63104693abbdd739bca159c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:15.922982 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4feb96efdfc44fe4a7cd46dc8deb4fe7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:15.943234 19025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b54311f4269d4575b8998d7f545000e7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:15.928417 18989 leader_election.cc:343] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 1df475e5bf1b4a54afeb0839831e0ac4
W20250111 02:57:15.950804 18842 leader_election.cc:343] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:15.951294 18842 leader_election.cc:304] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
W20250111 02:57:15.952441 18842 leader_election.cc:343] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:15.952855 18842 leader_election.cc:304] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:15.953773 19403 raft_consensus.cc:2743] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:15.954484 19395 raft_consensus.cc:2743] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:15.882035 19321 ts_tablet_manager.cc:1462] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa: Trace:
0111 02:57:14.602367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:14.611219 (+ 8852us) service_pool.cc:225] Handling call
0111 02:57:14.612776 (+ 1557us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0111 02:57:14.612857 (+ 81us) ts_tablet_manager.cc:676] Creating new metadata...
0111 02:57:14.627507 (+ 14650us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:14.660209 (+ 32702us) tablet_replica.cc:206] Creating consensus instance
0111 02:57:14.660542 (+ 333us) inbound_call.cc:173] Queueing success response
0111 02:57:15.082046 (+421504us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0111 02:57:15.285193 (+203147us) ts_tablet_manager.cc:1429] Starting tablet replica
0111 02:57:15.411605 (+126412us) tablet_replica.cc:260] Starting instrumentation
0111 02:57:15.452428 (+ 40823us) tablet_replica.cc:279] Starting consensus
Metrics: {"raft.queue_time_us":292509,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":11241,"tablet-open.queue_time_us":421564,"thread_start_us":296053,"threads_started":1}
I20250111 02:57:15.955780 18628 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet ede92343d9b54c29b8173a5eafcf68c6
I20250111 02:57:15.957384 18628 catalog_manager.cc:6596] ReplaceTablet: tablet ede92343d9b54c29b8173a5eafcf68c6 deleted and replaced by tablet 199ca8692f39445f93d8c27e2134a32c
I20250111 02:57:15.957445 19321 tablet_bootstrap.cc:492] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:15.946923 19329 consensus_queue.cc:237] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.969409 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet ede92343d9b54c29b8173a5eafcf68c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 199ca8692f39445f93d8c27e2134a32c) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:15.972222 18628 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_97b1974a358b42c694f22cd86099a2d6" } new_table_name: "default.Table_4139abd414944ffa86bf77e8875ad78c" modify_external_catalogs: true
I20250111 02:57:15.978166 19327 ts_tablet_manager.cc:1918] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:15.978574 19327 log.cc:1198] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/bfee658632fe46db8f3eb133198993a2
I20250111 02:57:15.983599 19392 leader_election.cc:290] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:15.984506 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6aa50a4dfe4398a82877b6dfcb5c29" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:15.985157 18910 raft_consensus.cc:2463] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:15.986419 19133 leader_election.cc:304] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:15.987257 19322 tablet_bootstrap.cc:492] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:15.987677 19322 ts_tablet_manager.cc:1397] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.300s user 0.027s sys 0.082s
I20250111 02:57:15.992117 19023 tablet_service.cc:1514] Processing DeleteTablet for tablet 616655aa75824fffa46f81c4f95d5159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:15.993667 19321 tablet_bootstrap.cc:654] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:15.993494 19322 raft_consensus.cc:357] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.994179 19322 raft_consensus.cc:383] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:15.994432 19322 raft_consensus.cc:738] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:15.995049 19322 consensus_queue.cc:260] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:15.999125 18626 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 3fa8530e894c498cbbfa73a3ba1c751e
I20250111 02:57:16.000136 19322 ts_tablet_manager.cc:1428] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250111 02:57:16.001175 18626 catalog_manager.cc:6596] ReplaceTablet: tablet 3fa8530e894c498cbbfa73a3ba1c751e deleted and replaced by tablet a983a670c2104398ac1a159ac3a1da7c
I20250111 02:57:16.002877 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa8530e894c498cbbfa73a3ba1c751e with delete_type TABLET_DATA_DELETED (replaced by tablet a983a670c2104398ac1a159ac3a1da7c) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:16.004392 18627 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 29561e984e6d40fea032075f0531dd1f
I20250111 02:57:16.003836 19023 tablet_service.cc:1467] Processing CreateTablet for tablet f1802e7349934fbb9d5f7eda37c3df20 (DEFAULT_TABLE table=default.Table_6883f75ce59a4deaa7ca2c46b1b87f9f [id=b2f59358b7c34e0b88ae3d165f2879b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.008107 19023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1802e7349934fbb9d5f7eda37c3df20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.008507 19327 ts_tablet_manager.cc:1939] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:16.008637 19392 raft_consensus.cc:2798] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:16.008987 19392 raft_consensus.cc:491] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:16.009308 19392 raft_consensus.cc:3054] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:16.009327 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet 29561e984e6d40fea032075f0531dd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:51082
W20250111 02:57:15.971375 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 349784us
I20250111 02:57:16.023677 19022 tablet_service.cc:1514] Processing DeleteTablet for tablet ede92343d9b54c29b8173a5eafcf68c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 199ca8692f39445f93d8c27e2134a32c) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:16.025529 19322 tablet_bootstrap.cc:492] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:16.027797 19022 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa8530e894c498cbbfa73a3ba1c751e with delete_type TABLET_DATA_DELETED (replaced by tablet a983a670c2104398ac1a159ac3a1da7c) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:16.029007 19022 tablet_service.cc:1514] Processing DeleteTablet for tablet 29561e984e6d40fea032075f0531dd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:15.996527 19027 tablet_service.cc:1467] Processing CreateTablet for tablet d8128b3941d54ce2a72bf2ae762a774f (DEFAULT_TABLE table=default.Table_8b6452ba06af45bab320b50842e527d8 [id=0fa5e1f97e984916a703c7b7da0f1087]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.030778 19027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8128b3941d54ce2a72bf2ae762a774f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.033856 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet bfee658632fe46db8f3eb133198993a2 (table default.Table_64ebd542bba841a3bd2e93366eef5bd3 [id=5deae041515148e091db2f4de7580ddf]) successfully deleted
W20250111 02:57:16.034740 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 149157c1847a497e88a3136a9c1983e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 149157c1847a497e88a3136a9c1983e9 already in progress: creating tablet
I20250111 02:57:16.034943 19327 tablet_replica.cc:331] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:16.035670 19327 raft_consensus.cc:2238] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:16.036134 19327 raft_consensus.cc:2267] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:16.037860 19054 raft_consensus.cc:2463] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:16.038543 19176 tablet_service.cc:1514] Processing DeleteTablet for tablet 616655aa75824fffa46f81c4f95d5159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:16.039182 18842 leader_election.cc:304] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters: d0cedab82b154d62b78809edc6f28bfa
W20250111 02:57:16.039886 18842 raft_consensus.cc:2681] T bfee658632fe46db8f3eb133198993a2 P 8221604733fe4fc0a1fc32de8d6af71c: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250111 02:57:16.040524 19392 raft_consensus.cc:513] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.041695 19327 ts_tablet_manager.cc:1905] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:16.043668 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6aa50a4dfe4398a82877b6dfcb5c29" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:16.044236 18910 raft_consensus.cc:3054] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:16.045022 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6aa50a4dfe4398a82877b6dfcb5c29" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:16.051609 18910 raft_consensus.cc:2463] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:16.052666 19169 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe4e60db4244a9a9696153b630c575b (DEFAULT_TABLE table=default.Table_064d4893c74f4870bc45572dd481c63c [id=b0a0ff2f4d554503a18d05e0ff51402d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.052743 19133 leader_election.cc:304] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:16.059432 19322 tablet_bootstrap.cc:654] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:16.060245 19400 leader_election.cc:290] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:16.061446 19401 raft_consensus.cc:2798] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:16.051601 19170 tablet_service.cc:1467] Processing CreateTablet for tablet 29561e984e6d40fea032075f0531dd1f (DEFAULT_TABLE table=default.Table_acc960ebdd02430ab5aa41a4f75f8578 [id=dc8029d5413f4e838d330383cf0b6f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.068982 19176 tablet_service.cc:1467] Processing CreateTablet for tablet b54311f4269d4575b8998d7f545000e7 (DEFAULT_TABLE table=default.Table_292c103f235f421db8dacf834cca2c53 [id=b091f6cd63104693abbdd739bca159c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.070201 19392 leader_election.cc:290] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:16.076483 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe4e60db4244a9a9696153b630c575b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.071164 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6aa50a4dfe4398a82877b6dfcb5c29" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:16.082157 19172 tablet_service.cc:1514] Processing DeleteTablet for tablet ede92343d9b54c29b8173a5eafcf68c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 199ca8692f39445f93d8c27e2134a32c) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:16.085386 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b54311f4269d4575b8998d7f545000e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.074368 19330 tablet_bootstrap.cc:654] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:16.077642 19401 raft_consensus.cc:695] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:16.089825 18627 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_8a622a8a40fa4d97899666729fdb7604" } new_table_name: "default.Table_211b4c873fa34d6b8d9f4c1a303d5b28" modify_external_catalogs: true
I20250111 02:57:16.093988 19322 tablet_bootstrap.cc:492] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:16.094425 19322 ts_tablet_manager.cc:1397] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.001s
I20250111 02:57:16.079561 19171 tablet_service.cc:1467] Processing CreateTablet for tablet f1802e7349934fbb9d5f7eda37c3df20 (DEFAULT_TABLE table=default.Table_6883f75ce59a4deaa7ca2c46b1b87f9f [id=b2f59358b7c34e0b88ae3d165f2879b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.081193 19177 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa8530e894c498cbbfa73a3ba1c751e with delete_type TABLET_DATA_DELETED (replaced by tablet a983a670c2104398ac1a159ac3a1da7c) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:16.081715 19166 tablet_service.cc:1514] Processing DeleteTablet for tablet 29561e984e6d40fea032075f0531dd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:16.096599 19401 consensus_queue.cc:237] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.097429 19322 raft_consensus.cc:357] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:16.098093 19322 raft_consensus.cc:383] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:16.098363 19322 raft_consensus.cc:738] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:16.100423 19022 tablet_service.cc:1514] Processing DeleteTablet for tablet bfee658632fe46db8f3eb133198993a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:16.099882 19345 raft_consensus.cc:513] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.104656 19053 raft_consensus.cc:2388] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0cedab82b154d62b78809edc6f28bfa in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
W20250111 02:57:16.108896 19029 consensus_meta.cc:287] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f: Time spent flushing consensus metadata: real 0.734s user 0.004s sys 0.004s
I20250111 02:57:16.087301 19170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29561e984e6d40fea032075f0531dd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.114739 18621 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_42756e535547448f9335dab7bd96f772"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:16.115417 19171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1802e7349934fbb9d5f7eda37c3df20. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:16.116556 18621 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42756e535547448f9335dab7bd96f772 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:16.117799 18626 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_39701f385f884aac9233a842fcaf93be" } modify_external_catalogs: true
I20250111 02:57:16.122594 19327 ts_tablet_manager.cc:1918] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:16.123112 19327 log.cc:1198] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/4feb96efdfc44fe4a7cd46dc8deb4fe7
I20250111 02:57:16.123962 19327 ts_tablet_manager.cc:1939] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:16.127307 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 (table default.Table_6883f75ce59a4deaa7ca2c46b1b87f9f [id=b2f59358b7c34e0b88ae3d165f2879b0]) successfully deleted
I20250111 02:57:16.127938 18626 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_39701f385f884aac9233a842fcaf93be" } modify_external_catalogs: true
I20250111 02:57:16.133100 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6aa50a4dfe4398a82877b6dfcb5c29" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:16.134199 18910 raft_consensus.cc:2388] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 54628fbcec2f4359aa9e26a734d4d29f in current term 1: Already voted for candidate d0cedab82b154d62b78809edc6f28bfa in this term.
W20250111 02:57:16.152168 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 616655aa75824fffa46f81c4f95d5159 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 616655aa75824fffa46f81c4f95d5159 already in progress: creating tablet
W20250111 02:57:16.158130 18627 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 066c87476a854e169fd4f5f1cdbb49f7 consensus state has no leader
W20250111 02:57:16.163610 18628 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 199ca8692f39445f93d8c27e2134a32c has no consensus state
I20250111 02:57:16.167411 19345 leader_election.cc:290] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:16.172422 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet ede92343d9b54c29b8173a5eafcf68c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ede92343d9b54c29b8173a5eafcf68c6 already in progress: creating tablet
W20250111 02:57:16.174430 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 3fa8530e894c498cbbfa73a3ba1c751e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fa8530e894c498cbbfa73a3ba1c751e already in progress: creating tablet
W20250111 02:57:16.176299 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 29561e984e6d40fea032075f0531dd1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29561e984e6d40fea032075f0531dd1f already in progress: creating tablet
I20250111 02:57:16.098969 19322 consensus_queue.cc:260] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:16.182085 19322 ts_tablet_manager.cc:1428] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.087s user 0.002s sys 0.003s
I20250111 02:57:16.185526 19321 tablet_bootstrap.cc:492] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:16.185966 19321 ts_tablet_manager.cc:1397] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.229s user 0.040s sys 0.034s
I20250111 02:57:16.188427 19321 raft_consensus.cc:357] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.189543 19321 raft_consensus.cc:383] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:16.189846 19321 raft_consensus.cc:738] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:16.190445 19321 consensus_queue.cc:260] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.192078 19167 tablet_service.cc:1467] Processing CreateTablet for tablet d8128b3941d54ce2a72bf2ae762a774f (DEFAULT_TABLE table=default.Table_8b6452ba06af45bab320b50842e527d8 [id=0fa5e1f97e984916a703c7b7da0f1087]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.199327 18628 master_service.cc:946] ReplaceTablet: received request to replace tablet b54311f4269d4575b8998d7f545000e7 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:16.205932 19322 tablet_bootstrap.cc:492] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:16.209933 18627 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_24a171ea51bb46648f255e69a57f2b36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:16.211815 18627 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24a171ea51bb46648f255e69a57f2b36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:16.217247 18629 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_680b04dc2c944efca139c2561bd52a54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:16.221016 18625 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_8b6452ba06af45bab320b50842e527d8" } modify_external_catalogs: true
I20250111 02:57:16.221699 18625 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_8b6452ba06af45bab320b50842e527d8" } modify_external_catalogs: true
W20250111 02:57:16.223363 18629 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_680b04dc2c944efca139c2561bd52a54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:16.224926 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_e2df5efa4e624d0fa2c7c164dd44f826"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:16.226711 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2df5efa4e624d0fa2c7c164dd44f826 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:16.237771 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8128b3941d54ce2a72bf2ae762a774f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.239425 19322 tablet_bootstrap.cc:654] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:16.239843 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6aa50a4dfe4398a82877b6dfcb5c29" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:16.240289 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df475e5bf1b4a54afeb0839831e0ac4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:16.240885 19196 raft_consensus.cc:2463] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:16.242367 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6aa50a4dfe4398a82877b6dfcb5c29" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:16.248327 18986 leader_election.cc:304] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:16.249480 18986 leader_election.cc:304] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f; no voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:16.250567 19345 raft_consensus.cc:2798] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:16.250890 19345 raft_consensus.cc:491] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:16.251248 19345 raft_consensus.cc:3054] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:16.255350 19400 raft_consensus.cc:2743] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250111 02:57:16.256618 18621 master_service.cc:946] ReplaceTablet: received request to replace tablet b715e32011114e6286c0b8e5406258c4 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:16.260409 19345 raft_consensus.cc:513] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.262952 19345 leader_election.cc:290] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:16.294718 19329 raft_consensus.cc:491] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:16.295287 19329 raft_consensus.cc:513] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:16.297255 19329 leader_election.cc:290] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:16.298794 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5baef8688ede42fca28cfb6b0bdeefa6" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:16.300274 18841 leader_election.cc:343] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:16.307940 19325 ts_tablet_manager.cc:1918] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:16.309039 19325 log.cc:1198] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/38433855438c47318dd579c8fe3bdf7f
W20250111 02:57:16.309335 18989 leader_election.cc:343] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 1df475e5bf1b4a54afeb0839831e0ac4
I20250111 02:57:16.309906 19325 ts_tablet_manager.cc:1939] T 38433855438c47318dd579c8fe3bdf7f P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:16.310601 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5baef8688ede42fca28cfb6b0bdeefa6" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:16.317690 18614 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 616655aa75824fffa46f81c4f95d5159 (table default.Table_a58de727de864b9a8ecab09048b72fb7 [id=581224f40f0c4f03a9224f06d660c883], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 616655aa75824fffa46f81c4f95d5159 consensus state has no leader
W20250111 02:57:16.318580 18842 leader_election.cc:343] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:16.322803 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df475e5bf1b4a54afeb0839831e0ac4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:16.323659 19196 raft_consensus.cc:3054] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:16.319025 18842 leader_election.cc:304] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:16.324820 19329 raft_consensus.cc:2743] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:16.333938 19196 raft_consensus.cc:2463] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:16.336913 19321 ts_tablet_manager.cc:1428] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.151s user 0.007s sys 0.000s
I20250111 02:57:16.338021 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 17) took 1889 ms. Trace:
I20250111 02:57:16.338568 19325 rpcz_store.cc:276] 0111 02:57:14.448556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:14.448723 (+ 167us) service_pool.cc:225] Handling call
0111 02:57:15.594755 (+1146032us) ts_tablet_manager.cc:1145] Deleting tablet 38433855438c47318dd579c8fe3bdf7f
0111 02:57:15.594804 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:16.220578 (+625774us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:16.307728 (+ 87150us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:16.337917 (+ 30189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23054,"tablet-delete.queue_time_us":1145329}
I20250111 02:57:16.345021 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 18) took 1843 ms. Trace:
I20250111 02:57:16.345566 19325 rpcz_store.cc:276] 0111 02:57:14.502073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:14.503115 (+ 1042us) service_pool.cc:225] Handling call
0111 02:57:16.344465 (+1841350us) ts_tablet_manager.cc:1145] Deleting tablet bfee658632fe46db8f3eb133198993a2
0111 02:57:16.344532 (+ 67us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:16.344936 (+ 404us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1840654}
I20250111 02:57:16.364099 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 38433855438c47318dd579c8fe3bdf7f (table default.Table_5876a73b13bf4596ba282611078406f6 [id=3fb54c7f24824b61ac0aea5abe813383]) successfully deleted
W20250111 02:57:16.364828 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet bfee658632fe46db8f3eb133198993a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfee658632fe46db8f3eb133198993a2 already in progress: creating tablet
I20250111 02:57:16.365674 18986 leader_election.cc:304] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:16.367051 19322 tablet_bootstrap.cc:492] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:16.338322 19321 tablet_bootstrap.cc:492] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:16.367509 19322 ts_tablet_manager.cc:1397] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.185s user 0.036s sys 0.023s
I20250111 02:57:16.369426 18622 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_0d373957ef8548a88268ff1eaf56f6a3" } new_table_name: "default.Table_053b33f398ad460e8e4cae9a9a80b13d" modify_external_catalogs: true
I20250111 02:57:16.370783 19345 raft_consensus.cc:2798] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:16.371290 19322 raft_consensus.cc:357] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:16.371685 19345 raft_consensus.cc:695] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:16.371984 19322 raft_consensus.cc:383] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:16.372232 19322 raft_consensus.cc:738] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:16.372903 19322 consensus_queue.cc:260] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:16.372918 19345 consensus_queue.cc:237] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.382524 19322 ts_tablet_manager.cc:1428] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250111 02:57:16.386171 19322 tablet_bootstrap.cc:492] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:16.389398 19321 tablet_bootstrap.cc:654] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:16.392429 19322 tablet_bootstrap.cc:654] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:16.396886 19326 ts_tablet_manager.cc:1918] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:16.347009 19325 tablet_replica.cc:331] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:16.423303 19325 raft_consensus.cc:2238] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:16.425827 18627 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_1c2c853d9f8446d4aa94f39f9eeb1d9f" } new_table_name: "default.Table_7ffc93386dff41da8bbdaf9f883bc27b" modify_external_catalogs: true
I20250111 02:57:16.426930 19027 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df475e5bf1b4a54afeb0839831e0ac4 with delete_type TABLET_DATA_DELETED (replaced by tablet 616655aa75824fffa46f81c4f95d5159) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:16.443128 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_03f0b101f0c64274a40b43897fe6dcdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:16.444934 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03f0b101f0c64274a40b43897fe6dcdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:16.446295 19321 tablet_bootstrap.cc:492] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:16.446847 19321 ts_tablet_manager.cc:1397] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.109s user 0.006s sys 0.005s
I20250111 02:57:16.450011 19321 raft_consensus.cc:357] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.438244 19325 raft_consensus.cc:2267] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:16.457684 19325 ts_tablet_manager.cc:1905] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:16.453922 19330 tablet_bootstrap.cc:492] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:16.470222 19330 ts_tablet_manager.cc:1397] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.645s user 0.016s sys 0.036s
I20250111 02:57:16.450992 19321 raft_consensus.cc:383] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:16.471527 19321 raft_consensus.cc:738] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:16.442907 19326 log.cc:1198] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/9bfd35c596c34aee9327cb017fecb4a7
I20250111 02:57:16.479470 19321 consensus_queue.cc:260] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.483026 19326 ts_tablet_manager.cc:1939] T 9bfd35c596c34aee9327cb017fecb4a7 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:16.474121 19330 raft_consensus.cc:357] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.488659 19330 raft_consensus.cc:383] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:16.490056 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 9bfd35c596c34aee9327cb017fecb4a7 (table default.Table_44c452b417f8496a89679757f4f4846e [id=e7a2df18562b4471bcb289ca08bf6574]) successfully deleted
W20250111 02:57:16.490747 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 149157c1847a497e88a3136a9c1983e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 149157c1847a497e88a3136a9c1983e9 already in progress: creating tablet
I20250111 02:57:16.489835 19330 raft_consensus.cc:738] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
W20250111 02:57:16.491370 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 already in progress: creating tablet
I20250111 02:57:16.492955 19330 consensus_queue.cc:260] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.490773 19321 ts_tablet_manager.cc:1428] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.044s user 0.007s sys 0.001s
I20250111 02:57:16.494406 19322 tablet_bootstrap.cc:492] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
W20250111 02:57:16.494926 18622 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 94b9d14959df49a6bbd80ba9e9e57ee4 has no consensus state
I20250111 02:57:16.494894 19321 tablet_bootstrap.cc:492] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:16.497856 19322 ts_tablet_manager.cc:1397] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.112s user 0.048s sys 0.050s
I20250111 02:57:16.500149 19329 raft_consensus.cc:491] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:16.500631 19329 raft_consensus.cc:513] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:16.501520 19322 raft_consensus.cc:357] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:16.502178 19322 raft_consensus.cc:383] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:16.502422 19322 raft_consensus.cc:738] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:16.503159 19329 leader_election.cc:290] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:16.502992 19322 consensus_queue.cc:260] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:16.505064 19322 ts_tablet_manager.cc:1428] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250111 02:57:16.504904 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f610a4ff1cc34a928f2db73f14834b25" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:16.505932 19322 tablet_bootstrap.cc:492] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
W20250111 02:57:16.506718 18842 leader_election.cc:343] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:16.509429 19326 tablet_replica.cc:331] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:16.511018 19326 raft_consensus.cc:2238] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:16.511554 19326 raft_consensus.cc:2267] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:16.513394 19326 ts_tablet_manager.cc:1905] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:16.515197 19330 ts_tablet_manager.cc:1428] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.044s user 0.006s sys 0.001s
I20250111 02:57:16.518932 19330 tablet_bootstrap.cc:492] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:16.520888 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f610a4ff1cc34a928f2db73f14834b25" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:16.522406 18841 leader_election.cc:343] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:16.522866 18841 leader_election.cc:304] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:16.529981 19329 raft_consensus.cc:2743] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:16.539784 19322 tablet_bootstrap.cc:654] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:16.545149 19170 tablet_service.cc:1467] Processing CreateTablet for tablet c28958a1018344a48d18e6c0f03cd0a3 (DEFAULT_TABLE table=default.Table_7bf5032295a34c379d00c2d9ad7e417f [id=f6f9076d18654f339a6feac0c468e265]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.547518 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 0dbb63ab78f84995ae6d9fb2aae78dcf (DEFAULT_TABLE table=default.Table_154c30276c8d4a8e805dc2fb85867bbf [id=9d3552a625804f0a9350454f3b5dc2e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.548105 19321 tablet_bootstrap.cc:654] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:16.549930 18886 tablet_service.cc:1467] Processing CreateTablet for tablet c28958a1018344a48d18e6c0f03cd0a3 (DEFAULT_TABLE table=default.Table_7bf5032295a34c379d00c2d9ad7e417f [id=f6f9076d18654f339a6feac0c468e265]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.551589 18886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c28958a1018344a48d18e6c0f03cd0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.553149 18630 catalog_manager.cc:5526] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65). New cstate: current_term: 1 leader_uuid: "8221604733fe4fc0a1fc32de8d6af71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: UNKNOWN } } }
I20250111 02:57:16.554301 19023 tablet_service.cc:1467] Processing CreateTablet for tablet 0dbb63ab78f84995ae6d9fb2aae78dcf (DEFAULT_TABLE table=default.Table_154c30276c8d4a8e805dc2fb85867bbf [id=9d3552a625804f0a9350454f3b5dc2e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.553479 19326 ts_tablet_manager.cc:1918] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:16.556866 19326 log.cc:1198] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/bfee658632fe46db8f3eb133198993a2
I20250111 02:57:16.557092 19023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dbb63ab78f84995ae6d9fb2aae78dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.557514 19326 ts_tablet_manager.cc:1939] T bfee658632fe46db8f3eb133198993a2 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:16.546945 19170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c28958a1018344a48d18e6c0f03cd0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.560444 19027 tablet_service.cc:1467] Processing CreateTablet for tablet c28958a1018344a48d18e6c0f03cd0a3 (DEFAULT_TABLE table=default.Table_7bf5032295a34c379d00c2d9ad7e417f [id=f6f9076d18654f339a6feac0c468e265]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.561555 19027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c28958a1018344a48d18e6c0f03cd0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.562451 19322 tablet_bootstrap.cc:492] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:16.562884 19322 ts_tablet_manager.cc:1397] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.002s
I20250111 02:57:16.565423 19322 raft_consensus.cc:357] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:16.566416 19322 raft_consensus.cc:383] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:16.566679 19322 raft_consensus.cc:738] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:16.567512 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dbb63ab78f84995ae6d9fb2aae78dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.567293 19322 consensus_queue.cc:260] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:16.568670 19326 tablet_replica.cc:331] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:16.569388 19326 raft_consensus.cc:2238] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:16.571517 19322 ts_tablet_manager.cc:1428] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20250111 02:57:16.572160 19326 pending_rounds.cc:62] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f: Trying to abort 1 pending ops.
I20250111 02:57:16.572466 19326 pending_rounds.cc:69] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112967112289927168 op_type: NO_OP noop_request { }
I20250111 02:57:16.572978 19326 raft_consensus.cc:2883] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:57:16.573330 19326 raft_consensus.cc:2267] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:16.574415 19322 tablet_bootstrap.cc:492] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:16.574649 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 0dbb63ab78f84995ae6d9fb2aae78dcf (DEFAULT_TABLE table=default.Table_154c30276c8d4a8e805dc2fb85867bbf [id=9d3552a625804f0a9350454f3b5dc2e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.575817 18878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dbb63ab78f84995ae6d9fb2aae78dcf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:16.575913 19417 consensus_queue.cc:1159] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "54628fbcec2f4359aa9e26a734d4d29f" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
W20250111 02:57:16.579519 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 616655aa75824fffa46f81c4f95d5159 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 616655aa75824fffa46f81c4f95d5159 already in progress: creating tablet
W20250111 02:57:16.580186 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet ede92343d9b54c29b8173a5eafcf68c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ede92343d9b54c29b8173a5eafcf68c6 already in progress: creating tablet
I20250111 02:57:16.580104 19169 tablet_service.cc:1467] Processing CreateTablet for tablet ef70b417568049f0b496f4d8e5f7e0e4 (DEFAULT_TABLE table=default.Table_a4c544f93b2a4ad5afd4aa78f3989b6f [id=cf9df41fb95c4e9c89cf0f52217843a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.581274 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef70b417568049f0b496f4d8e5f7e0e4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:16.580845 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 3fa8530e894c498cbbfa73a3ba1c751e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fa8530e894c498cbbfa73a3ba1c751e already in progress: creating tablet
I20250111 02:57:16.586102 19167 tablet_service.cc:1467] Processing CreateTablet for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (DEFAULT_TABLE table=default.Table_1c2c853d9f8446d4aa94f39f9eeb1d9f [id=b9b226a7c0da4da3aca14ddcba6ab5fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.587235 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ef3f45e5784a29b3b32cc0de6bac84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.587996 18623 catalog_manager.cc:5526] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa reported cstate change: term changed from 0 to 1, leader changed from <none> to d0cedab82b154d62b78809edc6f28bfa (127.18.20.67). New cstate: current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:16.589574 18623 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]). Expected version 4 got 0
I20250111 02:57:16.593955 19326 ts_tablet_manager.cc:1905] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:16.600503 19330 tablet_bootstrap.cc:654] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:16.600531 18880 tablet_service.cc:1467] Processing CreateTablet for tablet ef70b417568049f0b496f4d8e5f7e0e4 (DEFAULT_TABLE table=default.Table_a4c544f93b2a4ad5afd4aa78f3989b6f [id=cf9df41fb95c4e9c89cf0f52217843a5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250111 02:57:16.591502 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 29561e984e6d40fea032075f0531dd1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29561e984e6d40fea032075f0531dd1f already in progress: creating tablet
I20250111 02:57:16.601694 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef70b417568049f0b496f4d8e5f7e0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.601703 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet bfee658632fe46db8f3eb133198993a2 (table default.Table_64ebd542bba841a3bd2e93366eef5bd3 [id=5deae041515148e091db2f4de7580ddf]) successfully deleted
I20250111 02:57:16.618261 18879 tablet_service.cc:1467] Processing CreateTablet for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (DEFAULT_TABLE table=default.Table_1c2c853d9f8446d4aa94f39f9eeb1d9f [id=b9b226a7c0da4da3aca14ddcba6ab5fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.620085 18879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ef3f45e5784a29b3b32cc0de6bac84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.594854 19325 ts_tablet_manager.cc:1918] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:16.622782 19325 log.cc:1198] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/9bfd35c596c34aee9327cb017fecb4a7
I20250111 02:57:16.623113 19322 tablet_bootstrap.cc:654] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:16.623528 19325 ts_tablet_manager.cc:1939] T 9bfd35c596c34aee9327cb017fecb4a7 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:16.628785 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_61c40191c611402da3829f2ef5379f61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:16.630189 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 22) took 1973 ms. Trace:
W20250111 02:57:16.630628 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61c40191c611402da3829f2ef5379f61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:16.636564 18627 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8ef3f45e5784a29b3b32cc0de6bac84 consensus state has no leader
I20250111 02:57:16.638549 18626 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 5118d4b9cfbb49e8af6af56180858336
I20250111 02:57:16.642369 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet 5118d4b9cfbb49e8af6af56180858336 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:16.643324 18625 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet d8128b3941d54ce2a72bf2ae762a774f
I20250111 02:57:16.630538 19325 rpcz_store.cc:276] 0111 02:57:14.657373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:14.657580 (+ 207us) service_pool.cc:225] Handling call
0111 02:57:16.346793 (+1689213us) ts_tablet_manager.cc:1145] Deleting tablet 9bfd35c596c34aee9327cb017fecb4a7
0111 02:57:16.346840 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:16.495197 (+148357us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:16.594607 (+ 99410us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:16.630120 (+ 35513us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22652,"tablet-delete.queue_time_us":1688188}
W20250111 02:57:16.644651 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 5118d4b9cfbb49e8af6af56180858336 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5118d4b9cfbb49e8af6af56180858336 already in progress: creating tablet
I20250111 02:57:16.645081 19325 tablet_replica.cc:331] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:16.646442 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet d8128b3941d54ce2a72bf2ae762a774f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:51082
W20250111 02:57:16.648803 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet d8128b3941d54ce2a72bf2ae762a774f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8128b3941d54ce2a72bf2ae762a774f already in progress: creating tablet
I20250111 02:57:16.649626 19325 raft_consensus.cc:2238] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:16.650795 19325 raft_consensus.cc:2267] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:16.655079 19322 tablet_bootstrap.cc:492] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:16.656085 19322 ts_tablet_manager.cc:1397] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.002s
I20250111 02:57:16.658608 18621 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet b715e32011114e6286c0b8e5406258c4
I20250111 02:57:16.658921 19322 raft_consensus.cc:357] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:16.657213 19325 ts_tablet_manager.cc:1905] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:16.659852 19024 tablet_service.cc:1467] Processing CreateTablet for tablet ef70b417568049f0b496f4d8e5f7e0e4 (DEFAULT_TABLE table=default.Table_a4c544f93b2a4ad5afd4aa78f3989b6f [id=cf9df41fb95c4e9c89cf0f52217843a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.660290 18621 catalog_manager.cc:6596] ReplaceTablet: tablet b715e32011114e6286c0b8e5406258c4 deleted and replaced by tablet 8414653def014ce98bc38d87aac75bae
I20250111 02:57:16.661759 19024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef70b417568049f0b496f4d8e5f7e0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.663733 19171 tablet_service.cc:1514] Processing DeleteTablet for tablet 5118d4b9cfbb49e8af6af56180858336 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:16.663774 19033 tablet_service.cc:1467] Processing CreateTablet for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (DEFAULT_TABLE table=default.Table_1c2c853d9f8446d4aa94f39f9eeb1d9f [id=b9b226a7c0da4da3aca14ddcba6ab5fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:16.664646 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet d8128b3941d54ce2a72bf2ae762a774f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:16.664880 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet b715e32011114e6286c0b8e5406258c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8414653def014ce98bc38d87aac75bae) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:16.665066 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 9bfd35c596c34aee9327cb017fecb4a7 (table default.Table_44c452b417f8496a89679757f4f4846e [id=e7a2df18562b4471bcb289ca08bf6574]) successfully deleted
I20250111 02:57:16.665364 19345 raft_consensus.cc:491] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:16.665808 19345 raft_consensus.cc:513] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.671396 19345 leader_election.cc:290] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:16.672686 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2234755d474740a55184660119eb11" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:16.673219 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_c2fa721b96ba4e9ca293ff4b5c455447"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:16.674422 18986 leader_election.cc:343] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:16.674990 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2fa721b96ba4e9ca293ff4b5c455447 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:16.675359 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ef3f45e5784a29b3b32cc0de6bac84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:16.676285 19321 tablet_bootstrap.cc:492] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:16.676014 18621 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_bf50e4c0392544f8a24b861d3edaf46e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:16.676716 19321 ts_tablet_manager.cc:1397] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.182s user 0.010s sys 0.006s
W20250111 02:57:16.678490 18621 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf50e4c0392544f8a24b861d3edaf46e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:16.682672 18630 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet 5baef8688ede42fca28cfb6b0bdeefa6 (table default.Table_7f4578da9f5e418c9d67a535471b45cd [id=154c37dc0dba44bdab5e0315756dc7cf]). Expected version 1 got 0
I20250111 02:57:16.687485 19025 tablet_service.cc:1514] Processing DeleteTablet for tablet 5118d4b9cfbb49e8af6af56180858336 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:16.689157 19321 raft_consensus.cc:357] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.689608 19025 tablet_service.cc:1514] Processing DeleteTablet for tablet d8128b3941d54ce2a72bf2ae762a774f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:16.689874 19321 raft_consensus.cc:383] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:16.690131 19321 raft_consensus.cc:738] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:16.690757 19321 consensus_queue.cc:260] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.692772 19321 ts_tablet_manager.cc:1428] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.016s user 0.004s sys 0.002s
I20250111 02:57:16.693399 19025 tablet_service.cc:1514] Processing DeleteTablet for tablet b715e32011114e6286c0b8e5406258c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8414653def014ce98bc38d87aac75bae) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:16.701597 19401 raft_consensus.cc:491] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:16.702075 19401 raft_consensus.cc:513] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.703999 19401 leader_election.cc:290] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:16.707652 19133 leader_election.cc:343] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: bfee658632fe46db8f3eb133198993a2
W20250111 02:57:16.709854 19131 leader_election.cc:343] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: bfee658632fe46db8f3eb133198993a2
I20250111 02:57:16.710268 19131 leader_election.cc:304] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:16.710896 19325 ts_tablet_manager.cc:1918] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:16.711273 19325 log.cc:1198] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/1df475e5bf1b4a54afeb0839831e0ac4
I20250111 02:57:16.711547 19401 raft_consensus.cc:2743] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:16.712039 19325 ts_tablet_manager.cc:1939] T 1df475e5bf1b4a54afeb0839831e0ac4 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:16.715485 19326 ts_tablet_manager.cc:1918] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:16.670023 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2234755d474740a55184660119eb11" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:16.719426 18910 raft_consensus.cc:2463] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:16.695461 18880 tablet_service.cc:1514] Processing DeleteTablet for tablet 149157c1847a497e88a3136a9c1983e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:16.720734 19327 tablet_replica.cc:331] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:16.721402 19327 raft_consensus.cc:2238] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:16.721839 19327 raft_consensus.cc:2267] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:16.723853 19321 tablet_bootstrap.cc:492] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:16.725582 19327 ts_tablet_manager.cc:1905] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:16.726594 19329 raft_consensus.cc:491] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:16.728579 19329 raft_consensus.cc:513] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.729804 19321 tablet_bootstrap.cc:654] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:16.733306 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b715e32011114e6286c0b8e5406258c4" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:16.734278 19329 leader_election.cc:290] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:16.734525 18628 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet b54311f4269d4575b8998d7f545000e7
W20250111 02:57:16.735563 18841 leader_election.cc:343] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:16.738379 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 24) took 1892 ms. Trace:
I20250111 02:57:16.738973 19325 rpcz_store.cc:276] 0111 02:57:14.846458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:14.855655 (+ 9197us) service_pool.cc:225] Handling call
0111 02:57:16.644928 (+1789273us) ts_tablet_manager.cc:1145] Deleting tablet 1df475e5bf1b4a54afeb0839831e0ac4
0111 02:57:16.644971 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:16.703715 (+ 58744us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:16.710686 (+ 6971us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:16.738285 (+ 27599us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1778665}
I20250111 02:57:16.741693 18628 catalog_manager.cc:6596] ReplaceTablet: tablet b54311f4269d4575b8998d7f545000e7 deleted and replaced by tablet 845da3ee0863430ea3fe9de8c84f2c17
I20250111 02:57:16.742344 19169 tablet_service.cc:1514] Processing DeleteTablet for tablet b54311f4269d4575b8998d7f545000e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 845da3ee0863430ea3fe9de8c84f2c17) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:16.742625 18880 tablet_service.cc:1514] Processing DeleteTablet for tablet b54311f4269d4575b8998d7f545000e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 845da3ee0863430ea3fe9de8c84f2c17) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:16.743762 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 34) took 1481 ms. Trace:
I20250111 02:57:16.744995 19329 raft_consensus.cc:491] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:16.745518 19329 raft_consensus.cc:513] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:16.748201 19329 leader_election.cc:290] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:16.744146 19325 rpcz_store.cc:276] 0111 02:57:15.262069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:15.263291 (+ 1222us) service_pool.cc:225] Handling call
0111 02:57:16.743305 (+1480014us) ts_tablet_manager.cc:1145] Deleting tablet 72bcbe140a82462d909bd6d41bfbd2af
0111 02:57:16.743359 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:16.743704 (+ 345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1477601}
W20250111 02:57:16.749549 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 72bcbe140a82462d909bd6d41bfbd2af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72bcbe140a82462d909bd6d41bfbd2af already in progress: creating tablet
I20250111 02:57:16.750824 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 35) took 1488 ms. Trace:
I20250111 02:57:16.751217 19325 rpcz_store.cc:276] 0111 02:57:15.262406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:15.262515 (+ 109us) service_pool.cc:225] Handling call
0111 02:57:16.750455 (+1487940us) ts_tablet_manager.cc:1145] Deleting tablet adf93d013fb84ada9ff86ad964273e00
0111 02:57:16.750512 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:16.750765 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1479016}
W20250111 02:57:16.752465 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet adf93d013fb84ada9ff86ad964273e00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adf93d013fb84ada9ff86ad964273e00 already in progress: creating tablet
I20250111 02:57:16.752562 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5148fa2e6504561bd1b1a19e48993d7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:16.753997 18841 leader_election.cc:343] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:16.709849 18886 tablet_service.cc:1514] Processing DeleteTablet for tablet b715e32011114e6286c0b8e5406258c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8414653def014ce98bc38d87aac75bae) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:16.755950 19322 raft_consensus.cc:383] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:16.756261 19322 raft_consensus.cc:738] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:16.756284 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 39) took 1247 ms. Trace:
I20250111 02:57:16.757016 19325 rpcz_store.cc:276] 0111 02:57:15.506888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:15.507171 (+ 283us) service_pool.cc:225] Handling call
0111 02:57:16.753314 (+1246143us) ts_tablet_manager.cc:1145] Deleting tablet 149157c1847a497e88a3136a9c1983e9
0111 02:57:16.753358 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:16.753602 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1245386}
I20250111 02:57:16.757100 19322 consensus_queue.cc:260] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:16.762521 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 149157c1847a497e88a3136a9c1983e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 149157c1847a497e88a3136a9c1983e9 already in progress: creating tablet
I20250111 02:57:16.762801 18623 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_4139abd414944ffa86bf77e8875ad78c" } new_table_name: "default.Table_b48d4141dcc04405a8797e7f7ccd3f53" modify_external_catalogs: true
I20250111 02:57:16.766156 19322 ts_tablet_manager.cc:1428] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.110s user 0.005s sys 0.000s
I20250111 02:57:16.767270 19322 tablet_bootstrap.cc:492] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:16.770676 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 41) took 1247 ms. Trace:
I20250111 02:57:16.771826 19325 rpcz_store.cc:276] 0111 02:57:15.522933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:15.523141 (+ 208us) service_pool.cc:225] Handling call
0111 02:57:16.770305 (+1247164us) ts_tablet_manager.cc:1145] Deleting tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7
0111 02:57:16.770344 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:16.770613 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1212597}
W20250111 02:57:16.774793 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 already in progress: creating tablet
W20250111 02:57:16.777351 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 616655aa75824fffa46f81c4f95d5159 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 616655aa75824fffa46f81c4f95d5159 already in progress: creating tablet
W20250111 02:57:16.782105 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet ede92343d9b54c29b8173a5eafcf68c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ede92343d9b54c29b8173a5eafcf68c6 already in progress: creating tablet
W20250111 02:57:16.790409 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 29561e984e6d40fea032075f0531dd1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29561e984e6d40fea032075f0531dd1f already in progress: creating tablet
I20250111 02:57:16.791332 19329 consensus_queue.cc:1035] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250111 02:57:16.792359 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 3fa8530e894c498cbbfa73a3ba1c751e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fa8530e894c498cbbfa73a3ba1c751e already in progress: creating tablet
W20250111 02:57:16.794128 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 5118d4b9cfbb49e8af6af56180858336 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5118d4b9cfbb49e8af6af56180858336 already in progress: creating tablet
W20250111 02:57:16.800828 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet d8128b3941d54ce2a72bf2ae762a774f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8128b3941d54ce2a72bf2ae762a774f already in progress: creating tablet
I20250111 02:57:16.723569 18989 leader_election.cc:304] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters: d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:16.746999 19025 tablet_service.cc:1514] Processing DeleteTablet for tablet b54311f4269d4575b8998d7f545000e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 845da3ee0863430ea3fe9de8c84f2c17) from {username='slave'} at 127.0.0.1:36190
W20250111 02:57:16.802594 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 79144.1us
I20250111 02:57:16.803033 19321 tablet_bootstrap.cc:492] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:16.803480 19321 ts_tablet_manager.cc:1397] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.080s user 0.006s sys 0.004s
I20250111 02:57:16.804311 19345 raft_consensus.cc:2798] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:16.804674 19345 raft_consensus.cc:491] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:16.804984 19345 raft_consensus.cc:3054] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:16.772699 19330 tablet_bootstrap.cc:492] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:16.806177 19330 ts_tablet_manager.cc:1397] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.290s user 0.009s sys 0.002s
I20250111 02:57:16.807433 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b715e32011114e6286c0b8e5406258c4" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:16.808506 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5148fa2e6504561bd1b1a19e48993d7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:16.808655 19330 raft_consensus.cc:357] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.809243 19330 raft_consensus.cc:383] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:16.809505 19330 raft_consensus.cc:738] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:16.810045 19330 consensus_queue.cc:260] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:16.810608 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet b715e32011114e6286c0b8e5406258c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b715e32011114e6286c0b8e5406258c4 already in progress: creating tablet
W20250111 02:57:16.811261 18842 consensus_peers.cc:487] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c -> Peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Couldn't send request to peer 54628fbcec2f4359aa9e26a734d4d29f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:16.812608 18842 leader_election.cc:343] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:16.812884 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet b54311f4269d4575b8998d7f545000e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b54311f4269d4575b8998d7f545000e7 already in progress: creating tablet
I20250111 02:57:16.813197 18842 leader_election.cc:304] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
W20250111 02:57:16.814275 18842 leader_election.cc:343] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:16.814745 18842 leader_election.cc:304] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:16.823485 19427 raft_consensus.cc:2743] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:16.823263 19345 raft_consensus.cc:513] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.824237 19427 raft_consensus.cc:2743] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:16.806587 19321 raft_consensus.cc:357] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.824805 19321 raft_consensus.cc:383] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:16.825134 19321 raft_consensus.cc:738] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:16.825907 19321 consensus_queue.cc:260] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.826911 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2234755d474740a55184660119eb11" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:16.827536 18910 raft_consensus.cc:3054] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:16.829085 19330 ts_tablet_manager.cc:1428] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20250111 02:57:16.839258 19321 ts_tablet_manager.cc:1428] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.036s user 0.008s sys 0.000s
I20250111 02:57:16.840060 18910 raft_consensus.cc:2463] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:16.841043 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_c4faffc9659848ea9bff581e8d88d40e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:16.843322 19321 tablet_bootstrap.cc:492] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
W20250111 02:57:16.842797 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4faffc9659848ea9bff581e8d88d40e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:16.845134 19327 ts_tablet_manager.cc:1918] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:16.845556 19327 log.cc:1198] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/149157c1847a497e88a3136a9c1983e9
I20250111 02:57:16.846210 19327 ts_tablet_manager.cc:1939] T 149157c1847a497e88a3136a9c1983e9 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:16.858408 19321 tablet_bootstrap.cc:654] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:16.830047 19330 tablet_bootstrap.cc:492] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:16.864717 19322 tablet_bootstrap.cc:654] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:16.841324 18989 leader_election.cc:304] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:16.865533 19400 raft_consensus.cc:2798] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:16.841686 19326 log.cc:1198] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/1df475e5bf1b4a54afeb0839831e0ac4
I20250111 02:57:16.866899 19326 ts_tablet_manager.cc:1939] T 1df475e5bf1b4a54afeb0839831e0ac4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:16.874058 19345 leader_election.cc:290] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:16.875339 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a2234755d474740a55184660119eb11" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:16.875999 19197 raft_consensus.cc:3054] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:16.877900 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 1df475e5bf1b4a54afeb0839831e0ac4 (table default.Table_a58de727de864b9a8ecab09048b72fb7 [id=581224f40f0c4f03a9224f06d660c883]) successfully deleted
W20250111 02:57:16.878584 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 5118d4b9cfbb49e8af6af56180858336 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5118d4b9cfbb49e8af6af56180858336 already in progress: creating tablet
W20250111 02:57:16.879187 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet d8128b3941d54ce2a72bf2ae762a774f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8128b3941d54ce2a72bf2ae762a774f already in progress: creating tablet
I20250111 02:57:16.887760 19197 raft_consensus.cc:2463] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:16.890684 19321 tablet_bootstrap.cc:492] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:16.890540 18630 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_680b04dc2c944efca139c2561bd52a54" } new_table_name: "default.Table_9ace2d46973441a5add29e9eeacbe6a7" modify_external_catalogs: true
I20250111 02:57:16.891120 19321 ts_tablet_manager.cc:1397] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.000s
I20250111 02:57:16.890995 18626 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_f731216a607f4fdaadd2e8dbedf7552d" } modify_external_catalogs: true
I20250111 02:57:16.891579 18626 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_f731216a607f4fdaadd2e8dbedf7552d" } modify_external_catalogs: true
I20250111 02:57:16.891726 19345 raft_consensus.cc:491] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:16.892139 19345 raft_consensus.cc:513] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.895321 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfad6c08b4b44c1a664cda3b43acbb1" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:16.899881 19321 raft_consensus.cc:357] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:16.900633 19321 raft_consensus.cc:383] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:16.900908 19321 raft_consensus.cc:738] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:16.901365 19327 tablet_replica.cc:331] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:16.901557 19321 consensus_queue.cc:260] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:16.902122 19327 raft_consensus.cc:2238] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:16.902582 19327 raft_consensus.cc:2267] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:16.903162 19400 raft_consensus.cc:695] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:16.904253 19400 consensus_queue.cc:237] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.909611 19345 leader_election.cc:290] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:16.911616 19330 tablet_bootstrap.cc:654] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:16.905305 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 149157c1847a497e88a3136a9c1983e9 (table default.Table_d76c4a77f7d244478357d5a5fc3cdda6 [id=b46d11174b994b818737139b9b2eb13a]) successfully deleted
I20250111 02:57:16.909260 19321 ts_tablet_manager.cc:1428] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
W20250111 02:57:16.913064 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet b54311f4269d4575b8998d7f545000e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b54311f4269d4575b8998d7f545000e7 already in progress: creating tablet
I20250111 02:57:16.913628 19321 tablet_bootstrap.cc:492] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:16.913957 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfad6c08b4b44c1a664cda3b43acbb1" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:16.917886 18986 leader_election.cc:304] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f; no voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:16.919548 19345 raft_consensus.cc:2743] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:16.933099 19321 tablet_bootstrap.cc:654] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:16.938624 19330 tablet_bootstrap.cc:492] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
W20250111 02:57:16.890619 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet b715e32011114e6286c0b8e5406258c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b715e32011114e6286c0b8e5406258c4 already in progress: creating tablet
I20250111 02:57:16.939514 19330 ts_tablet_manager.cc:1397] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.110s user 0.012s sys 0.001s
W20250111 02:57:16.939836 18615 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 49251.5us
W20250111 02:57:16.940531 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet b54311f4269d4575b8998d7f545000e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b54311f4269d4575b8998d7f545000e7 already in progress: creating tablet
I20250111 02:57:16.945166 18627 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_211b4c873fa34d6b8d9f4c1a303d5b28" } modify_external_catalogs: true
I20250111 02:57:16.945791 18627 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_211b4c873fa34d6b8d9f4c1a303d5b28" } modify_external_catalogs: true
I20250111 02:57:16.952282 18622 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_902d917b19634ad5b20fadd455c0ba88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:16.954068 18622 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_902d917b19634ad5b20fadd455c0ba88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:16.955080 19327 ts_tablet_manager.cc:1905] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:16.967655 18623 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 199ca8692f39445f93d8c27e2134a32c has no consensus state
W20250111 02:57:16.974011 18630 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92aa64b2de5746d1b12cc93a10c9d56e has no consensus state
I20250111 02:57:16.973798 19330 raft_consensus.cc:357] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.974522 19330 raft_consensus.cc:383] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:16.974769 19330 raft_consensus.cc:738] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:16.976938 19322 tablet_bootstrap.cc:492] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:16.976537 19330 consensus_queue.cc:260] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:16.977355 19322 ts_tablet_manager.cc:1397] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.210s user 0.011s sys 0.000s
I20250111 02:57:16.978554 19330 ts_tablet_manager.cc:1428] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.039s user 0.003s sys 0.002s
I20250111 02:57:16.978932 19054 raft_consensus.cc:1270] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Refusing update from remote peer d0cedab82b154d62b78809edc6f28bfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:57:16.980808 18630 master_service.cc:946] ReplaceTablet: received request to replace tablet f610a4ff1cc34a928f2db73f14834b25 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:16.980553 19401 consensus_queue.cc:1035] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:16.988950 18910 raft_consensus.cc:1270] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Refusing update from remote peer d0cedab82b154d62b78809edc6f28bfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:57:16.990923 19401 consensus_queue.cc:1035] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:16.979732 19322 raft_consensus.cc:357] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:16.995396 19322 raft_consensus.cc:383] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:16.995702 19322 raft_consensus.cc:738] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:16.996335 19322 consensus_queue.cc:260] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.002306 19322 ts_tablet_manager.cc:1428] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20250111 02:57:17.003248 19322 tablet_bootstrap.cc:492] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:17.007359 19327 ts_tablet_manager.cc:1918] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:17.007771 19327 log.cc:1198] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/b715e32011114e6286c0b8e5406258c4
I20250111 02:57:17.009018 19327 ts_tablet_manager.cc:1939] T b715e32011114e6286c0b8e5406258c4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:17.010520 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet b715e32011114e6286c0b8e5406258c4 (table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]) successfully deleted
I20250111 02:57:17.012149 19330 tablet_bootstrap.cc:492] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:17.016495 18622 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_2e1ef9ed4749430f848ae883c239e519"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:17.018754 18622 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e1ef9ed4749430f848ae883c239e519 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:17.021401 19330 tablet_bootstrap.cc:654] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:17.024950 18623 master_service.cc:946] ReplaceTablet: received request to replace tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:17.025919 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_4672fdb5296042f3b9373401c98fa509"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:17.026618 19321 tablet_bootstrap.cc:492] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:17.027002 19321 ts_tablet_manager.cc:1397] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.114s user 0.006s sys 0.003s
I20250111 02:57:17.044741 19321 raft_consensus.cc:357] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.045471 19321 raft_consensus.cc:383] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250111 02:57:17.045529 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4672fdb5296042f3b9373401c98fa509 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:17.045727 19321 raft_consensus.cc:738] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:17.052666 18624 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]). Expected version 4 got 0
I20250111 02:57:17.061935 19433 mvcc.cc:204] Tried to move back new op lower bound from 7112967114635939840 to 7112967111742439424. Current Snapshot: MvccSnapshot[applied={T|T < 7112967114635939840}]
I20250111 02:57:17.072144 19169 tablet_service.cc:1514] Processing DeleteTablet for tablet bfee658632fe46db8f3eb133198993a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:17.072952 19325 tablet_replica.cc:331] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:17.073498 18627 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 066c87476a854e169fd4f5f1cdbb49f7
I20250111 02:57:17.073765 19325 raft_consensus.cc:2238] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:17.074324 19325 raft_consensus.cc:2267] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:17.075812 19027 tablet_service.cc:1514] Processing DeleteTablet for tablet 066c87476a854e169fd4f5f1cdbb49f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:17.076233 19325 ts_tablet_manager.cc:1905] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:17.076647 19442 tablet.cc:1721] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250111 02:57:17.077225 19441 tablet.cc:1721] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250111 02:57:17.078116 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 066c87476a854e169fd4f5f1cdbb49f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 066c87476a854e169fd4f5f1cdbb49f7 already in progress: creating tablet
I20250111 02:57:17.081368 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet 066c87476a854e169fd4f5f1cdbb49f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:17.080766 19321 consensus_queue.cc:260] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:17.082914 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 066c87476a854e169fd4f5f1cdbb49f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 066c87476a854e169fd4f5f1cdbb49f7 already in progress: creating tablet
I20250111 02:57:17.085067 19169 tablet_service.cc:1514] Processing DeleteTablet for tablet 066c87476a854e169fd4f5f1cdbb49f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:17.090993 19321 ts_tablet_manager.cc:1428] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.064s user 0.004s sys 0.001s
I20250111 02:57:17.094413 19330 tablet_bootstrap.cc:492] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:17.094848 19330 ts_tablet_manager.cc:1397] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.083s user 0.015s sys 0.002s
I20250111 02:57:17.096267 19321 tablet_bootstrap.cc:492] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:17.098899 19345 raft_consensus.cc:491] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:17.099442 19345 raft_consensus.cc:513] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:17.101754 19345 leader_election.cc:290] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:17.106390 18989 leader_election.cc:343] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 4feb96efdfc44fe4a7cd46dc8deb4fe7
I20250111 02:57:17.107153 19330 raft_consensus.cc:357] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.107826 19330 raft_consensus.cc:383] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:17.108074 19330 raft_consensus.cc:738] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:17.108742 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4feb96efdfc44fe4a7cd46dc8deb4fe7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:17.108675 19330 consensus_queue.cc:260] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.109486 19197 raft_consensus.cc:2463] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:17.110886 18986 leader_election.cc:304] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:17.124056 19330 ts_tablet_manager.cc:1428] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20250111 02:57:17.125481 19345 raft_consensus.cc:2798] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:17.125944 19345 raft_consensus.cc:491] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:17.126353 19345 raft_consensus.cc:3054] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:17.133530 19330 tablet_bootstrap.cc:492] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:17.122917 19321 tablet_bootstrap.cc:654] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:17.140831 19027 tablet_service.cc:1467] Processing CreateTablet for tablet 92aa64b2de5746d1b12cc93a10c9d56e (DEFAULT_TABLE table=default.Table_9ace2d46973441a5add29e9eeacbe6a7 [id=d2dff624ad604b4c8746d5ba6ed87de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.141997 19027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92aa64b2de5746d1b12cc93a10c9d56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.143972 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 92aa64b2de5746d1b12cc93a10c9d56e (DEFAULT_TABLE table=default.Table_9ace2d46973441a5add29e9eeacbe6a7 [id=d2dff624ad604b4c8746d5ba6ed87de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.144807 19169 tablet_service.cc:1467] Processing CreateTablet for tablet 92aa64b2de5746d1b12cc93a10c9d56e (DEFAULT_TABLE table=default.Table_9ace2d46973441a5add29e9eeacbe6a7 [id=d2dff624ad604b4c8746d5ba6ed87de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.145901 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92aa64b2de5746d1b12cc93a10c9d56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.145182 18878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92aa64b2de5746d1b12cc93a10c9d56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.158699 19330 tablet_bootstrap.cc:654] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:17.164404 19321 tablet_bootstrap.cc:492] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:17.164862 19321 ts_tablet_manager.cc:1397] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.069s user 0.006s sys 0.006s
I20250111 02:57:17.171555 19024 tablet_service.cc:1467] Processing CreateTablet for tablet 40fdec982cf643c68ef90798f4f65ad4 (DEFAULT_TABLE table=default.Table_28852db6fe0d4d5bae2db5ad7fa3ac63 [id=c594b9f9a14b4f97949baf078e58b927]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.172780 19024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40fdec982cf643c68ef90798f4f65ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.175341 19345 raft_consensus.cc:513] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:17.175892 19321 raft_consensus.cc:357] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.174933 18626 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 5fe8aa967fcb4bd7a48f6f0f69bea496
I20250111 02:57:17.176609 19321 raft_consensus.cc:383] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:17.176872 19321 raft_consensus.cc:738] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:17.177313 19345 leader_election.cc:290] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:17.177465 19321 consensus_queue.cc:260] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.179267 18886 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:51082
W20250111 02:57:17.181210 18617 catalog_manager.cc:4839] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 because the tablet was not found. No further retry: Not found: Tablet not found: 5fe8aa967fcb4bd7a48f6f0f69bea496
W20250111 02:57:17.184559 18989 leader_election.cc:343] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 4feb96efdfc44fe4a7cd46dc8deb4fe7
I20250111 02:57:17.185518 18629 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_2130e8dde62a424788eb31bb9f19c2e0" } new_table_name: "default.Table_c8ee4857085f45dfb5adc86d85bead9d" modify_external_catalogs: true
I20250111 02:57:17.187872 18621 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_9ace2d46973441a5add29e9eeacbe6a7" } new_table_name: "default.Table_04353d7ad63649d4bf1946d138f8b553" modify_external_catalogs: true
I20250111 02:57:17.190461 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:17.191866 19321 ts_tablet_manager.cc:1428] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
W20250111 02:57:17.192359 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 because the tablet was not found. No further retry: Not found: Tablet not found: 5fe8aa967fcb4bd7a48f6f0f69bea496
I20250111 02:57:17.192826 19321 tablet_bootstrap.cc:492] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:17.200304 18623 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 0d6aa50a4dfe4398a82877b6dfcb5c29
I20250111 02:57:17.202275 18886 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a03880a8b0e46b7858ed1c2c2dddcaa) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:17.203008 19327 tablet_replica.cc:331] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:17.203972 19327 raft_consensus.cc:2238] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:17.204715 19327 raft_consensus.cc:2267] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:17.215687 18623 catalog_manager.cc:6596] ReplaceTablet: tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 deleted and replaced by tablet 6a03880a8b0e46b7858ed1c2c2dddcaa
I20250111 02:57:17.216194 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a03880a8b0e46b7858ed1c2c2dddcaa) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:17.216921 19326 tablet_replica.cc:331] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:17.220595 19321 tablet_bootstrap.cc:654] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:17.220805 19176 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:17.222169 19176 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a03880a8b0e46b7858ed1c2c2dddcaa) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:17.223843 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4feb96efdfc44fe4a7cd46dc8deb4fe7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:17.224642 19197 raft_consensus.cc:3054] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:17.235996 19197 raft_consensus.cc:2463] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:17.236853 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_71ceb23d653647089e7bd2fa3da23c01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:17.238914 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71ceb23d653647089e7bd2fa3da23c01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:17.243557 18886 tablet_service.cc:1514] Processing DeleteTablet for tablet 72bcbe140a82462d909bd6d41bfbd2af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:17.244637 19327 ts_tablet_manager.cc:1905] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:17.244971 19330 tablet_bootstrap.cc:492] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:17.245404 19330 ts_tablet_manager.cc:1397] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.112s user 0.018s sys 0.007s
I20250111 02:57:17.247998 19330 raft_consensus.cc:357] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.248714 19330 raft_consensus.cc:383] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:17.248975 19330 raft_consensus.cc:738] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:17.255570 19330 consensus_queue.cc:260] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.258562 19330 ts_tablet_manager.cc:1428] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.013s user 0.003s sys 0.003s
I20250111 02:57:17.263453 18886 tablet_service.cc:1467] Processing CreateTablet for tablet 40fdec982cf643c68ef90798f4f65ad4 (DEFAULT_TABLE table=default.Table_28852db6fe0d4d5bae2db5ad7fa3ac63 [id=c594b9f9a14b4f97949baf078e58b927]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.265210 18886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40fdec982cf643c68ef90798f4f65ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.267349 19330 tablet_bootstrap.cc:492] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:17.288774 19322 tablet_bootstrap.cc:654] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:17.295089 19330 tablet_bootstrap.cc:654] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:17.310117 19330 tablet_bootstrap.cc:492] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:17.310546 19330 ts_tablet_manager.cc:1397] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.004s
I20250111 02:57:17.313645 19330 raft_consensus.cc:357] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.314368 19330 raft_consensus.cc:383] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:17.314632 19330 raft_consensus.cc:738] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:17.317492 19330 consensus_queue.cc:260] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.325505 19330 ts_tablet_manager.cc:1428] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20250111 02:57:17.326793 19330 tablet_bootstrap.cc:492] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:17.240509 18626 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_6823aef149894f49822fe303c036446d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:17.342528 18626 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6823aef149894f49822fe303c036446d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:17.346379 19330 tablet_bootstrap.cc:654] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:17.351613 18614 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 92aa64b2de5746d1b12cc93a10c9d56e (table default.Table_9ace2d46973441a5add29e9eeacbe6a7 [id=d2dff624ad604b4c8746d5ba6ed87de3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92aa64b2de5746d1b12cc93a10c9d56e consensus state has no leader
I20250111 02:57:17.351919 19027 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2377893bfc426fa00d5ddacd9c5961 (DEFAULT_TABLE table=default.Table_03f0b101f0c64274a40b43897fe6dcdb [id=d5b01ba62b1a42e7a1dca7769742db0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.353080 19027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b2377893bfc426fa00d5ddacd9c5961. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.331133 18628 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_f05e5d6100bc46379e8598ce5d367e03" } modify_external_catalogs: true
I20250111 02:57:17.363613 18628 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_f05e5d6100bc46379e8598ce5d367e03" } modify_external_catalogs: true
I20250111 02:57:17.308578 19427 raft_consensus.cc:491] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:17.379482 19427 raft_consensus.cc:513] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.381721 19427 leader_election.cc:290] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:17.383589 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72bcbe140a82462d909bd6d41bfbd2af" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:17.384259 19053 raft_consensus.cc:2463] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:17.385668 18842 leader_election.cc:304] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:17.387096 19427 raft_consensus.cc:2798] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:17.387742 19427 raft_consensus.cc:491] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:17.388338 19427 raft_consensus.cc:3054] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:17.394992 19326 raft_consensus.cc:2238] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:17.398887 18880 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2377893bfc426fa00d5ddacd9c5961 (DEFAULT_TABLE table=default.Table_03f0b101f0c64274a40b43897fe6dcdb [id=d5b01ba62b1a42e7a1dca7769742db0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.400172 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b2377893bfc426fa00d5ddacd9c5961. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.400938 18622 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_7bf5032295a34c379d00c2d9ad7e417f" } modify_external_catalogs: true
I20250111 02:57:17.401450 19330 tablet_bootstrap.cc:492] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:17.401554 18622 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_7bf5032295a34c379d00c2d9ad7e417f" } modify_external_catalogs: true
I20250111 02:57:17.401916 19330 ts_tablet_manager.cc:1397] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.000s
I20250111 02:57:17.408818 18879 tablet_service.cc:1467] Processing CreateTablet for tablet 94b9d14959df49a6bbd80ba9e9e57ee4 (DEFAULT_TABLE table=default.Table_053b33f398ad460e8e4cae9a9a80b13d [id=a0260905798243e9a870e755912fb0b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.409984 18879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b9d14959df49a6bbd80ba9e9e57ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.429059 19345 raft_consensus.cc:491] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:17.429569 19345 raft_consensus.cc:513] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.430737 19330 raft_consensus.cc:357] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:17.431437 19330 raft_consensus.cc:383] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:17.431545 19345 leader_election.cc:290] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:17.431710 19330 raft_consensus.cc:738] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:17.432324 19330 consensus_queue.cc:260] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:17.433830 19326 pending_rounds.cc:62] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f: Trying to abort 2 pending ops.
I20250111 02:57:17.435247 19024 tablet_service.cc:1467] Processing CreateTablet for tablet 94b9d14959df49a6bbd80ba9e9e57ee4 (DEFAULT_TABLE table=default.Table_053b33f398ad460e8e4cae9a9a80b13d [id=a0260905798243e9a870e755912fb0b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.436492 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5baef8688ede42fca28cfb6b0bdeefa6" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:17.437323 18909 raft_consensus.cc:2463] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:17.439875 19024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b9d14959df49a6bbd80ba9e9e57ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.442734 19330 ts_tablet_manager.cc:1428] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.041s user 0.009s sys 0.000s
I20250111 02:57:17.445755 19330 tablet_bootstrap.cc:492] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:17.451151 18989 leader_election.cc:304] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:17.453382 19400 raft_consensus.cc:2798] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:17.453930 19400 raft_consensus.cc:491] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:17.454388 19400 raft_consensus.cc:3054] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:17.238906 19321 tablet_bootstrap.cc:492] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:17.336695 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 40fdec982cf643c68ef90798f4f65ad4 (DEFAULT_TABLE table=default.Table_28852db6fe0d4d5bae2db5ad7fa3ac63 [id=c594b9f9a14b4f97949baf078e58b927]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.455451 19321 ts_tablet_manager.cc:1397] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.263s user 0.009s sys 0.003s
I20250111 02:57:17.434058 19326 pending_rounds.cc:69] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7112967114635939840 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "0d6aa50a4dfe4398a82877b6dfcb5c29" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 4 new_table_name: "default.Table_0158103117c842e0ae88e8f041658357" dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" new_extra_config { } }
I20250111 02:57:17.455936 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40fdec982cf643c68ef90798f4f65ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.456758 19326 pending_rounds.cc:69] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112967111742439424 op_type: NO_OP noop_request { }
I20250111 02:57:17.457363 19326 raft_consensus.cc:2883] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:57:17.457873 19326 raft_consensus.cc:2267] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:17.457846 19321 raft_consensus.cc:357] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.458561 19321 raft_consensus.cc:383] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:17.369642 19440 raft_consensus.cc:491] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:17.459089 19321 raft_consensus.cc:738] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:17.459286 19440 raft_consensus.cc:513] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.459715 19321 consensus_queue.cc:260] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.461243 19440 leader_election.cc:290] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:17.388711 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72bcbe140a82462d909bd6d41bfbd2af" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:17.461894 19196 raft_consensus.cc:2463] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:17.412882 19167 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2377893bfc426fa00d5ddacd9c5961 (DEFAULT_TABLE table=default.Table_03f0b101f0c64274a40b43897fe6dcdb [id=d5b01ba62b1a42e7a1dca7769742db0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.463397 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b2377893bfc426fa00d5ddacd9c5961. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.463249 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72bcbe140a82462d909bd6d41bfbd2af" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:17.465070 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5baef8688ede42fca28cfb6b0bdeefa6" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:17.419199 19170 tablet_service.cc:1467] Processing CreateTablet for tablet 94b9d14959df49a6bbd80ba9e9e57ee4 (DEFAULT_TABLE table=default.Table_053b33f398ad460e8e4cae9a9a80b13d [id=a0260905798243e9a870e755912fb0b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.467803 19170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b9d14959df49a6bbd80ba9e9e57ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.469005 19326 ts_tablet_manager.cc:1905] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:17.443267 19131 consensus_peers.cc:487] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa -> Peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Couldn't send request to peer 54628fbcec2f4359aa9e26a734d4d29f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:17.475723 19345 raft_consensus.cc:491] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:17.476383 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72bcbe140a82462d909bd6d41bfbd2af" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:17.476208 19345 raft_consensus.cc:513] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:17.476967 19054 raft_consensus.cc:2463] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:17.478843 19131 leader_election.cc:304] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:17.480621 19440 raft_consensus.cc:2798] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:17.481211 19440 raft_consensus.cc:491] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:17.481676 19440 raft_consensus.cc:3054] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:17.461913 19321 ts_tablet_manager.cc:1428] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250111 02:57:17.502090 19345 leader_election.cc:290] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:17.503610 19321 tablet_bootstrap.cc:492] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:17.512410 19427 raft_consensus.cc:513] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.514039 18909 raft_consensus.cc:2388] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0cedab82b154d62b78809edc6f28bfa in current term 1: Already voted for candidate 8221604733fe4fc0a1fc32de8d6af71c in this term.
I20250111 02:57:17.520205 19330 tablet_bootstrap.cc:654] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:17.518923 19400 raft_consensus.cc:513] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.537007 19400 leader_election.cc:290] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:17.544917 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5baef8688ede42fca28cfb6b0bdeefa6" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:17.545549 18910 raft_consensus.cc:3054] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:17.548956 19330 tablet_bootstrap.cc:492] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:17.549394 19330 ts_tablet_manager.cc:1397] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.104s user 0.009s sys 0.001s
I20250111 02:57:17.551738 19330 raft_consensus.cc:357] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.552448 19330 raft_consensus.cc:383] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:17.552699 19330 raft_consensus.cc:738] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:17.553310 19330 consensus_queue.cc:260] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.554266 19427 leader_election.cc:290] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:17.562026 19322 tablet_bootstrap.cc:492] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:17.562486 19322 ts_tablet_manager.cc:1397] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.559s user 0.010s sys 0.000s
I20250111 02:57:17.565282 19322 raft_consensus.cc:357] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.565975 19322 raft_consensus.cc:383] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:17.566246 19322 raft_consensus.cc:738] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:17.567620 19330 ts_tablet_manager.cc:1428] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20250111 02:57:17.568526 19330 tablet_bootstrap.cc:492] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:17.569020 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72bcbe140a82462d909bd6d41bfbd2af" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
W20250111 02:57:17.575208 18986 leader_election.cc:343] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:17.577126 19327 ts_tablet_manager.cc:1918] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250111 02:57:17.577503 19327 log.cc:1198] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/0d6aa50a4dfe4398a82877b6dfcb5c29
I20250111 02:57:17.578159 19327 ts_tablet_manager.cc:1939] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:17.578984 19325 ts_tablet_manager.cc:1918] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:17.579362 19325 log.cc:1198] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/bfee658632fe46db8f3eb133198993a2
I20250111 02:57:17.578976 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_5f09ae109ece4cd780bf64aa3d7b35c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:17.579955 19325 ts_tablet_manager.cc:1939] T bfee658632fe46db8f3eb133198993a2 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
W20250111 02:57:17.580775 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f09ae109ece4cd780bf64aa3d7b35c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:17.583684 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5baef8688ede42fca28cfb6b0bdeefa6" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:17.584450 18986 leader_election.cc:304] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:17.585343 19451 raft_consensus.cc:2798] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:17.586158 19451 raft_consensus.cc:695] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:17.587029 19451 consensus_queue.cc:237] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:17.589184 18630 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet f610a4ff1cc34a928f2db73f14834b25
W20250111 02:57:17.590957 18986 leader_election.cc:343] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:17.566861 19322 consensus_queue.cc:260] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.591868 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b715e32011114e6286c0b8e5406258c4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:17.593515 19322 ts_tablet_manager.cc:1428] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20250111 02:57:17.594655 19027 tablet_service.cc:1514] Processing DeleteTablet for tablet f610a4ff1cc34a928f2db73f14834b25 with delete_type TABLET_DATA_DELETED (replaced by tablet 855dac47872d4c5fbf8f5e38b588c7fe) from {username='slave'} at 127.0.0.1:36190
W20250111 02:57:17.596215 18986 leader_election.cc:343] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:17.597043 18630 catalog_manager.cc:6596] ReplaceTablet: tablet f610a4ff1cc34a928f2db73f14834b25 deleted and replaced by tablet 855dac47872d4c5fbf8f5e38b588c7fe
I20250111 02:57:17.597930 18630 master_service.cc:946] ReplaceTablet: received request to replace tablet ef70b417568049f0b496f4d8e5f7e0e4 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:17.600008 19330 tablet_bootstrap.cc:654] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:17.605947 19321 tablet_bootstrap.cc:654] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:17.610762 18989 leader_election.cc:343] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: b715e32011114e6286c0b8e5406258c4
I20250111 02:57:17.610848 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72bcbe140a82462d909bd6d41bfbd2af" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:17.611316 18989 leader_election.cc:304] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f; no voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:17.611536 19054 raft_consensus.cc:3054] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:17.610762 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet f610a4ff1cc34a928f2db73f14834b25 with delete_type TABLET_DATA_DELETED (replaced by tablet 855dac47872d4c5fbf8f5e38b588c7fe) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:17.615294 19400 raft_consensus.cc:2743] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:17.616245 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]) successfully deleted
I20250111 02:57:17.619894 19330 tablet_bootstrap.cc:492] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:17.620337 19330 ts_tablet_manager.cc:1397] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.001s
I20250111 02:57:17.625496 19327 tablet_replica.cc:331] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:17.626433 19327 raft_consensus.cc:2238] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:17.627525 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet f610a4ff1cc34a928f2db73f14834b25 with delete_type TABLET_DATA_DELETED (replaced by tablet 855dac47872d4c5fbf8f5e38b588c7fe) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:17.623003 19330 raft_consensus.cc:357] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.634385 19327 raft_consensus.cc:2267] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:17.634752 19330 raft_consensus.cc:383] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:17.635035 19330 raft_consensus.cc:738] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:17.635684 19330 consensus_queue.cc:260] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.627899 19322 tablet_bootstrap.cc:492] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:17.632094 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 74592f80afab4513a9eb2eac351c4de5 (DEFAULT_TABLE table=default.Table_24a171ea51bb46648f255e69a57f2b36 [id=7c2fda5567344ad8b8034c9aabe10231]), partition=RANGE (key) PARTITION UNBOUNDED
W20250111 02:57:17.638103 18616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250111 02:57:17.637452 18878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74592f80afab4513a9eb2eac351c4de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.636230 19327 ts_tablet_manager.cc:1905] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:17.644315 19427 raft_consensus.cc:491] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:17.644956 19427 raft_consensus.cc:513] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.647910 19427 leader_election.cc:290] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:17.648770 19325 tablet_replica.cc:331] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:17.649598 19325 raft_consensus.cc:2238] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:17.651440 19330 ts_tablet_manager.cc:1428] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.031s user 0.005s sys 0.002s
I20250111 02:57:17.651187 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c02166674a140f29e9f655d09532ced" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:17.652324 19330 tablet_bootstrap.cc:492] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
W20250111 02:57:17.652743 18842 leader_election.cc:343] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:17.652686 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c02166674a140f29e9f655d09532ced" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:17.653506 19027 tablet_service.cc:1467] Processing CreateTablet for tablet 74592f80afab4513a9eb2eac351c4de5 (DEFAULT_TABLE table=default.Table_24a171ea51bb46648f255e69a57f2b36 [id=7c2fda5567344ad8b8034c9aabe10231]), partition=RANGE (key) PARTITION UNBOUNDED
W20250111 02:57:17.654095 18841 leader_election.cc:343] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:17.654584 18841 leader_election.cc:304] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:17.656750 19427 raft_consensus.cc:2743] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:17.659397 19027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74592f80afab4513a9eb2eac351c4de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.667200 19054 raft_consensus.cc:2463] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:17.668498 18842 leader_election.cc:304] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
W20250111 02:57:17.669190 18842 raft_consensus.cc:2681] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250111 02:57:17.671697 18910 raft_consensus.cc:2463] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:17.673151 19440 raft_consensus.cc:513] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.674455 19195 raft_consensus.cc:2388] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate d0cedab82b154d62b78809edc6f28bfa in this term.
I20250111 02:57:17.674710 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet bfee658632fe46db8f3eb133198993a2 (table default.Table_64ebd542bba841a3bd2e93366eef5bd3 [id=5deae041515148e091db2f4de7580ddf]) successfully deleted
W20250111 02:57:17.675943 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 066c87476a854e169fd4f5f1cdbb49f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 066c87476a854e169fd4f5f1cdbb49f7 already in progress: creating tablet
I20250111 02:57:17.681147 19444 raft_consensus.cc:491] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:17.681262 18989 leader_election.cc:304] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters: d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:17.682041 19451 raft_consensus.cc:2798] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:17.681617 19444 raft_consensus.cc:513] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.682489 19451 raft_consensus.cc:695] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:17.683315 19451 consensus_queue.cc:237] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.683946 19444 leader_election.cc:290] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:17.689468 19427 raft_consensus.cc:491] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:17.690138 19427 raft_consensus.cc:513] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.692926 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa8530e894c498cbbfa73a3ba1c751e" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:17.693701 18910 raft_consensus.cc:2463] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:17.684626 19033 tablet_service.cc:1467] Processing CreateTablet for tablet 0267a92995ea460786dc26813b5895cf (DEFAULT_TABLE table=default.Table_e2df5efa4e624d0fa2c7c164dd44f826 [id=e7ade2345b0845acb3ecdaa6f1ff6e7b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250111 02:57:17.695574 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 because the tablet was not found. No further retry: Not found: Tablet not found: 5fe8aa967fcb4bd7a48f6f0f69bea496
I20250111 02:57:17.688522 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa8530e894c498cbbfa73a3ba1c751e" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:17.697242 19131 leader_election.cc:343] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:17.698859 19133 leader_election.cc:304] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f
I20250111 02:57:17.700055 19444 raft_consensus.cc:2798] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:17.700395 19444 raft_consensus.cc:491] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:17.700709 19444 raft_consensus.cc:3054] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:17.702927 19401 raft_consensus.cc:491] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:17.703399 19401 raft_consensus.cc:513] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.705250 19401 leader_election.cc:290] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:17.711256 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf93d013fb84ada9ff86ad964273e00" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:17.711906 19054 raft_consensus.cc:2463] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:17.712462 19330 tablet_bootstrap.cc:654] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:17.712353 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf93d013fb84ada9ff86ad964273e00" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:17.712991 19131 leader_election.cc:304] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:17.713932 19401 raft_consensus.cc:2798] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:17.714395 19401 raft_consensus.cc:491] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:17.714722 19401 raft_consensus.cc:3054] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:17.713932 19133 leader_election.cc:343] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:17.715420 19400 raft_consensus.cc:491] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:17.715843 19400 raft_consensus.cc:513] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.716923 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 with delete_type TABLET_DATA_DELETED (replaced by tablet f1802e7349934fbb9d5f7eda37c3df20) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:17.696125 19322 tablet_bootstrap.cc:654] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:17.717777 19400 leader_election.cc:290] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:17.718251 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet adf93d013fb84ada9ff86ad964273e00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:17.719810 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 74592f80afab4513a9eb2eac351c4de5 (DEFAULT_TABLE table=default.Table_24a171ea51bb46648f255e69a57f2b36 [id=7c2fda5567344ad8b8034c9aabe10231]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.720842 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74592f80afab4513a9eb2eac351c4de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.719280 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72bcbe140a82462d909bd6d41bfbd2af" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:17.723196 19427 leader_election.cc:290] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:17.723692 18989 leader_election.cc:343] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 149157c1847a497e88a3136a9c1983e9
I20250111 02:57:17.724817 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa8530e894c498cbbfa73a3ba1c751e" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:17.725011 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa8530e894c498cbbfa73a3ba1c751e" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:17.726379 18842 leader_election.cc:343] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:17.727874 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72bcbe140a82462d909bd6d41bfbd2af" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:17.728389 18616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.126:45115, user_credentials={real_user=slave}} blocked reactor thread for 90311.4us
W20250111 02:57:17.729226 19133 leader_election.cc:343] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: Tablet not RUNNING: STOPPED
I20250111 02:57:17.726998 19440 leader_election.cc:290] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:17.735466 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0267a92995ea460786dc26813b5895cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.735802 19054 raft_consensus.cc:2388] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d0cedab82b154d62b78809edc6f28bfa in current term 1: Already voted for candidate 8221604733fe4fc0a1fc32de8d6af71c in this term.
I20250111 02:57:17.737197 19131 leader_election.cc:304] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:17.741405 19440 raft_consensus.cc:2743] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250111 02:57:17.768781 19444 raft_consensus.cc:513] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.774618 19195 raft_consensus.cc:2388] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate d0cedab82b154d62b78809edc6f28bfa in this term.
I20250111 02:57:17.777406 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa8530e894c498cbbfa73a3ba1c751e" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:17.778218 19326 ts_tablet_manager.cc:1918] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250111 02:57:17.778618 19326 log.cc:1198] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/0d6aa50a4dfe4398a82877b6dfcb5c29
I20250111 02:57:17.779513 19326 ts_tablet_manager.cc:1939] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:17.782491 19330 tablet_bootstrap.cc:492] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
W20250111 02:57:17.782907 19131 leader_election.cc:343] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:17.770792 19444 leader_election.cc:290] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:17.783380 19330 ts_tablet_manager.cc:1397] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.131s user 0.010s sys 0.002s
I20250111 02:57:17.786275 19330 raft_consensus.cc:357] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.786949 19330 raft_consensus.cc:383] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:17.787225 19330 raft_consensus.cc:738] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:17.789871 19330 consensus_queue.cc:260] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.790514 19321 tablet_bootstrap.cc:492] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:17.790951 19321 ts_tablet_manager.cc:1397] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.288s user 0.010s sys 0.002s
I20250111 02:57:17.792002 19330 ts_tablet_manager.cc:1428] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250111 02:57:17.793751 19321 raft_consensus.cc:357] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:17.797741 19321 raft_consensus.cc:383] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:17.798147 19321 raft_consensus.cc:738] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:17.798521 19024 tablet_service.cc:1467] Processing CreateTablet for tablet 199ca8692f39445f93d8c27e2134a32c (DEFAULT_TABLE table=default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.799790 19024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 199ca8692f39445f93d8c27e2134a32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.800724 19330 tablet_bootstrap.cc:492] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:17.800951 19170 tablet_service.cc:1467] Processing CreateTablet for tablet 0267a92995ea460786dc26813b5895cf (DEFAULT_TABLE table=default.Table_e2df5efa4e624d0fa2c7c164dd44f826 [id=e7ade2345b0845acb3ecdaa6f1ff6e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.801252 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149157c1847a497e88a3136a9c1983e9" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:17.801877 19195 raft_consensus.cc:2463] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:17.802148 19170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0267a92995ea460786dc26813b5895cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.806248 19167 tablet_service.cc:1467] Processing CreateTablet for tablet 199ca8692f39445f93d8c27e2134a32c (DEFAULT_TABLE table=default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.807575 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 199ca8692f39445f93d8c27e2134a32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.798895 19321 consensus_queue.cc:260] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:17.812654 19023 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 (DEFAULT_TABLE table=default.Table_f731216a607f4fdaadd2e8dbedf7552d [id=e77c4aa40ab44fb3bc98193e503be4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.818692 19325 raft_consensus.cc:2267] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:17.821260 19325 ts_tablet_manager.cc:1905] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:17.822122 19023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:17.833209 18629 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b5148fa2e6504561bd1b1a19e48993d7 consensus state has no leader
W20250111 02:57:17.837165 18621 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92aa64b2de5746d1b12cc93a10c9d56e consensus state has no leader
I20250111 02:57:17.841765 18986 leader_election.cc:304] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:17.843138 19400 raft_consensus.cc:2798] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:17.843571 19400 raft_consensus.cc:491] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:17.844416 19400 raft_consensus.cc:3054] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:17.857123 19330 tablet_bootstrap.cc:654] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:17.860174 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa8530e894c498cbbfa73a3ba1c751e" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:17.860806 18910 raft_consensus.cc:3054] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:17.863190 18841 leader_election.cc:304] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:17.865466 18886 tablet_service.cc:1467] Processing CreateTablet for tablet 0267a92995ea460786dc26813b5895cf (DEFAULT_TABLE table=default.Table_e2df5efa4e624d0fa2c7c164dd44f826 [id=e7ade2345b0845acb3ecdaa6f1ff6e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.866197 18879 tablet_service.cc:1467] Processing CreateTablet for tablet 199ca8692f39445f93d8c27e2134a32c (DEFAULT_TABLE table=default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.866645 18886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0267a92995ea460786dc26813b5895cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.868378 18880 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 (DEFAULT_TABLE table=default.Table_f731216a607f4fdaadd2e8dbedf7552d [id=e77c4aa40ab44fb3bc98193e503be4cf]), partition=RANGE (key) PARTITION UNBOUNDED
W20250111 02:57:17.857291 18614 catalog_manager.cc:4951] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): alter failed for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]): Illegal state: Tablet not RUNNING: STOPPING
I20250111 02:57:17.869652 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.873235 19326 tablet_replica.cc:331] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:17.873980 19326 raft_consensus.cc:2238] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:17.874469 19326 raft_consensus.cc:2267] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:17.874419 19451 consensus_queue.cc:1035] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:17.875669 19401 raft_consensus.cc:513] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.878437 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf93d013fb84ada9ff86ad964273e00" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:17.879038 19054 raft_consensus.cc:3054] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:17.883718 19401 leader_election.cc:290] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:17.883771 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]) successfully deleted
I20250111 02:57:17.884972 19330 tablet_bootstrap.cc:492] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:17.885672 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf93d013fb84ada9ff86ad964273e00" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:17.886096 18879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 199ca8692f39445f93d8c27e2134a32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.886829 19330 ts_tablet_manager.cc:1397] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.086s user 0.009s sys 0.008s
I20250111 02:57:17.888579 19326 ts_tablet_manager.cc:1905] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:17.890326 19330 raft_consensus.cc:357] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:17.891798 19133 leader_election.cc:343] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:17.826871 19321 ts_tablet_manager.cc:1428] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.036s user 0.005s sys 0.001s
I20250111 02:57:17.891016 19330 raft_consensus.cc:383] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:17.908298 19330 raft_consensus.cc:738] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:17.908957 19330 consensus_queue.cc:260] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:17.911264 19330 ts_tablet_manager.cc:1428] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20250111 02:57:17.912119 19330 tablet_bootstrap.cc:492] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:17.913960 19321 tablet_bootstrap.cc:492] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:17.927417 19330 tablet_bootstrap.cc:654] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:17.942866 19169 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 (DEFAULT_TABLE table=default.Table_f731216a607f4fdaadd2e8dbedf7552d [id=e77c4aa40ab44fb3bc98193e503be4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.950599 18629 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_a68194fdb909449f878695d09b81c753"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:17.956496 18629 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a68194fdb909449f878695d09b81c753 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:17.963346 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.949863 19322 tablet_bootstrap.cc:492] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:17.966028 19322 ts_tablet_manager.cc:1397] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.338s user 0.009s sys 0.017s
I20250111 02:57:17.857291 19027 tablet_service.cc:1467] Processing CreateTablet for tablet 9192160e931445f1acd207604192f759 (DEFAULT_TABLE table=default.Table_42756e535547448f9335dab7bd96f772 [id=1496848ec9594cad889312978436d9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.975030 18882 tablet_service.cc:1467] Processing CreateTablet for tablet 9192160e931445f1acd207604192f759 (DEFAULT_TABLE table=default.Table_42756e535547448f9335dab7bd96f772 [id=1496848ec9594cad889312978436d9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.984580 18910 raft_consensus.cc:2463] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:17.985177 19427 raft_consensus.cc:2743] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:17.993671 19054 raft_consensus.cc:2463] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:17.997542 19327 ts_tablet_manager.cc:1918] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:17.998310 19327 log.cc:1198] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/72bcbe140a82462d909bd6d41bfbd2af
I20250111 02:57:17.999096 19327 ts_tablet_manager.cc:1939] T 72bcbe140a82462d909bd6d41bfbd2af P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:18.005003 19027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9192160e931445f1acd207604192f759. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:18.007463 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9192160e931445f1acd207604192f759. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:17.979589 19174 tablet_service.cc:1467] Processing CreateTablet for tablet 9192160e931445f1acd207604192f759 (DEFAULT_TABLE table=default.Table_42756e535547448f9335dab7bd96f772 [id=1496848ec9594cad889312978436d9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:17.992172 19133 leader_election.cc:304] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f
I20250111 02:57:18.020673 19401 raft_consensus.cc:2798] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:18.021107 19401 raft_consensus.cc:695] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:18.021850 19401 consensus_queue.cc:237] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.001287 19321 tablet_bootstrap.cc:654] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:18.037061 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9192160e931445f1acd207604192f759. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:18.060088 18622 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet c28958a1018344a48d18e6c0f03cd0a3
I20250111 02:57:18.066109 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet c28958a1018344a48d18e6c0f03cd0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:17 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:18.066797 18628 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 6a2234755d474740a55184660119eb11
I20250111 02:57:18.072244 18624 master_service.cc:946] ReplaceTablet: received request to replace tablet 29561e984e6d40fea032075f0531dd1f from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:18.078835 19327 tablet_replica.cc:331] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:18.079453 19400 raft_consensus.cc:513] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.082150 19400 leader_election.cc:290] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:18.083148 19326 ts_tablet_manager.cc:1918] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:18.083505 19326 log.cc:1198] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/f610a4ff1cc34a928f2db73f14834b25
I20250111 02:57:18.083997 19326 ts_tablet_manager.cc:1939] T f610a4ff1cc34a928f2db73f14834b25 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:18.085510 19327 raft_consensus.cc:2238] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:18.086055 19327 raft_consensus.cc:2267] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:18.088908 19327 ts_tablet_manager.cc:1905] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:18.097410 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa8530e894c498cbbfa73a3ba1c751e with delete_type TABLET_DATA_DELETED (replaced by tablet a983a670c2104398ac1a159ac3a1da7c) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:18.098420 18880 tablet_service.cc:1514] Processing DeleteTablet for tablet c28958a1018344a48d18e6c0f03cd0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:17 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:18.099864 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 72bcbe140a82462d909bd6d41bfbd2af (table default.Table_8d238bdc5f53451aab66467655c6fb2b [id=f35e4781910c45ba93c35a24da2b83c1]) successfully deleted
I20250111 02:57:18.103478 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet 149157c1847a497e88a3136a9c1983e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:18.104632 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa8530e894c498cbbfa73a3ba1c751e with delete_type TABLET_DATA_DELETED (replaced by tablet a983a670c2104398ac1a159ac3a1da7c) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:18.105746 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet 72bcbe140a82462d909bd6d41bfbd2af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:18.106774 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a2234755d474740a55184660119eb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:17 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:18.118340 19401 raft_consensus.cc:491] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:18.118918 19401 raft_consensus.cc:513] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:18.123332 19401 leader_election.cc:290] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:18.130908 18628 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_6883f75ce59a4deaa7ca2c46b1b87f9f" } new_table_name: "default.Table_23038750b6334f158755d8f4c06190f8" modify_external_catalogs: true
I20250111 02:57:18.137449 18622 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_5f45eb0bdbd447c19a7a840a6cd9448d" } new_table_name: "default.Table_c4d2c7776f6c4ad880888a23b8774780" modify_external_catalogs: true
I20250111 02:57:18.153115 18886 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a2234755d474740a55184660119eb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:17 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:18.158046 18886 tablet_service.cc:1467] Processing CreateTablet for tablet a983a670c2104398ac1a159ac3a1da7c (DEFAULT_TABLE table=default.Table_5f45eb0bdbd447c19a7a840a6cd9448d [id=a37b721314134c86b39d1f80b7359f19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:18.125187 19174 tablet_service.cc:1467] Processing CreateTablet for tablet a983a670c2104398ac1a159ac3a1da7c (DEFAULT_TABLE table=default.Table_5f45eb0bdbd447c19a7a840a6cd9448d [id=a37b721314134c86b39d1f80b7359f19]), partition=RANGE (key) PARTITION UNBOUNDED
W20250111 02:57:18.128878 19133 leader_election.cc:343] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: b715e32011114e6286c0b8e5406258c4
I20250111 02:57:18.164531 19330 tablet_bootstrap.cc:492] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:18.164950 19330 ts_tablet_manager.cc:1397] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.253s user 0.085s sys 0.096s
I20250111 02:57:18.167315 19330 raft_consensus.cc:357] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.168144 19330 raft_consensus.cc:383] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:18.168500 19330 raft_consensus.cc:738] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:18.169034 19330 consensus_queue.cc:260] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.173049 19322 raft_consensus.cc:357] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.173698 19330 ts_tablet_manager.cc:1428] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250111 02:57:18.173840 19322 raft_consensus.cc:383] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:18.174108 19322 raft_consensus.cc:738] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:18.174634 19330 tablet_bootstrap.cc:492] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:18.183456 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a983a670c2104398ac1a159ac3a1da7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:18.203534 18886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a983a670c2104398ac1a159ac3a1da7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:18.216329 18630 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet ef70b417568049f0b496f4d8e5f7e0e4
I20250111 02:57:18.217723 18630 catalog_manager.cc:6596] ReplaceTablet: tablet ef70b417568049f0b496f4d8e5f7e0e4 deleted and replaced by tablet 7ea78feee64340e39404cba0f1bf04db
I20250111 02:57:18.222150 19176 tablet_service.cc:1514] Processing DeleteTablet for tablet ef70b417568049f0b496f4d8e5f7e0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7ea78feee64340e39404cba0f1bf04db) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:18.225872 18626 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47522 (request call id 117) took 1022 ms. Trace:
I20250111 02:57:18.226302 18626 rpcz_store.cc:276] 0111 02:57:17.203769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:17.214125 (+ 10356us) service_pool.cc:225] Handling call
0111 02:57:17.240267 (+ 26142us) spinlock_profiling.cc:243] Waited 25 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d53306 00007fd427ab7044 00007fd427ab6fd2 00007fd427ab6f01 00007fd427ab6e7c 00007fd427ab6132 00007fd425d79dfc 00007fd425d79236
0111 02:57:17.343626 (+103359us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 02:57:17.346093 (+ 2467us) catalog_manager.cc:2122] Created new table and tablet info
0111 02:57:17.622740 (+276647us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b5400021300. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd428348a81 00007fd42841c9bc 00007fd4284125dd 00007fd4283d3c5b 00007fd4283d3a34 00007fd429d843a1 00007fd429d84bc6 00007fd429c33575 00007fd429d5333f 00007fd427ab7044 00007fd427ab6fd2
0111 02:57:18.224905 (+602165us) catalog_manager.cc:2182] Wrote table and tablets to system table
0111 02:57:18.225356 (+ 451us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0111 02:57:18.225799 (+ 443us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:18.164250 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:18.164591 (+ 341us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:18.164610 (+ 19us) write_op.cc:435] Acquired schema lock
0111 02:57:18.164627 (+ 17us) tablet.cc:659] Decoding operations
0111 02:57:18.164868 (+ 241us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:18.164901 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:18.164920 (+ 19us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:18.165088 (+ 168us) tablet.cc:698] Row locks acquired
0111 02:57:18.165100 (+ 12us) write_op.cc:260] PREPARE: finished
0111 02:57:18.165206 (+ 106us) write_op.cc:270] Start()
0111 02:57:18.165281 (+ 75us) write_op.cc:276] Timestamp: P: 1736564238165147 usec, L: 0
0111 02:57:18.165296 (+ 15us) op_driver.cc:348] REPLICATION: starting
0111 02:57:18.165718 (+ 422us) log.cc:844] Serialized 433 byte log entry
0111 02:57:18.222299 (+ 56581us) op_driver.cc:464] REPLICATION: finished
0111 02:57:18.223033 (+ 734us) write_op.cc:301] APPLY: starting
0111 02:57:18.223110 (+ 77us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:18.223231 (+ 121us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:18.223323 (+ 92us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:18.223720 (+ 397us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:18.223862 (+ 142us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0111 02:57:18.223908 (+ 46us) write_op.cc:312] APPLY: finished
0111 02:57:18.224083 (+ 175us) log.cc:844] Serialized 29 byte log entry
0111 02:57:18.224262 (+ 179us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:18.224337 (+ 75us) write_op.cc:454] Released schema lock
0111 02:57:18.224367 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:18.224606 (+ 239us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:18.224628 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":512408320,"child_traces":[["op",{"apply.queue_time_us":580,"apply.run_cpu_time_us":1544,"apply.run_wall_time_us":1542,"num_ops":2,"prepare.queue_time_us":541272,"prepare.run_cpu_time_us":1668,"prepare.run_wall_time_us":1655,"replication_time_us":56901}]]}
I20250111 02:57:18.230383 19321 tablet_bootstrap.cc:492] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
W20250111 02:57:18.232359 18622 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a983a670c2104398ac1a159ac3a1da7c consensus state has no leader
I20250111 02:57:18.241545 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet ef70b417568049f0b496f4d8e5f7e0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7ea78feee64340e39404cba0f1bf04db) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:18.174714 19322 consensus_queue.cc:260] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.249457 19322 ts_tablet_manager.cc:1428] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.272s user 0.005s sys 0.000s
I20250111 02:57:18.252156 19322 tablet_bootstrap.cc:492] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
W20250111 02:57:18.272792 18614 catalog_manager.cc:4945] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): alter failed for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]),no further retry: Illegal state: Tablet not RUNNING: STOPPED
I20250111 02:57:18.275774 19025 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a2234755d474740a55184660119eb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:17 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:18.277529 19327 ts_tablet_manager.cc:1918] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:18.277953 19327 log.cc:1198] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/f610a4ff1cc34a928f2db73f14834b25
I20250111 02:57:18.278609 19327 ts_tablet_manager.cc:1939] T f610a4ff1cc34a928f2db73f14834b25 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:18.282473 19026 tablet_service.cc:1514] Processing DeleteTablet for tablet ef70b417568049f0b496f4d8e5f7e0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7ea78feee64340e39404cba0f1bf04db) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:18.285887 19030 tablet_service.cc:1467] Processing CreateTablet for tablet a983a670c2104398ac1a159ac3a1da7c (DEFAULT_TABLE table=default.Table_5f45eb0bdbd447c19a7a840a6cd9448d [id=a37b721314134c86b39d1f80b7359f19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:18.288165 19034 tablet_service.cc:1514] Processing DeleteTablet for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 with delete_type TABLET_DATA_DELETED (replaced by tablet f1802e7349934fbb9d5f7eda37c3df20) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:18.289047 19327 tablet_replica.cc:331] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:18.289958 19327 raft_consensus.cc:2238] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:18.287485 19031 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a03880a8b0e46b7858ed1c2c2dddcaa) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:18.290513 19327 raft_consensus.cc:2267] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:18.293510 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet c28958a1018344a48d18e6c0f03cd0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:17 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:18.295449 19400 raft_consensus.cc:491] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:18.296147 19054 raft_consensus.cc:3054] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:18.296366 19400 raft_consensus.cc:513] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.297339 19330 tablet_bootstrap.cc:654] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:18.298477 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b715e32011114e6286c0b8e5406258c4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:18.299465 19326 tablet_replica.cc:331] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:18.299513 19053 raft_consensus.cc:2463] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:18.300592 19326 raft_consensus.cc:2238] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:18.302615 19400 leader_election.cc:290] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:18.303530 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet f610a4ff1cc34a928f2db73f14834b25 (table default.Table_fcfe588eae964e7b9f5af53b5728e639 [id=f1373922978a4598b888e1863525942e]) successfully deleted
I20250111 02:57:18.304379 19327 ts_tablet_manager.cc:1905] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:18.304231 18626 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_23d632d4476c46b78d27619201589641"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:18.306254 18626 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23d632d4476c46b78d27619201589641 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:18.272171 18622 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_65af1591521e4974a28304ce51fe6fb1" } modify_external_catalogs: true
I20250111 02:57:18.307389 18622 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_65af1591521e4974a28304ce51fe6fb1" } modify_external_catalogs: true
I20250111 02:57:18.258034 19131 leader_election.cc:304] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:18.295447 19030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a983a670c2104398ac1a159ac3a1da7c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:18.315963 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 58033us
I20250111 02:57:18.316808 19131 leader_election.cc:304] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:18.267575 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149157c1847a497e88a3136a9c1983e9" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:18.318431 19195 raft_consensus.cc:3054] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:18.322154 19444 raft_consensus.cc:2798] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:18.323398 19401 raft_consensus.cc:2798] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:18.323768 19401 raft_consensus.cc:491] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:18.324081 19401 raft_consensus.cc:3054] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:18.328667 18627 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_71ceb23d653647089e7bd2fa3da23c01" } modify_external_catalogs: true
I20250111 02:57:18.329348 18627 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_71ceb23d653647089e7bd2fa3da23c01" } modify_external_catalogs: true
I20250111 02:57:18.311403 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5148fa2e6504561bd1b1a19e48993d7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:18.335193 18623 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet b8e8c72c99ac4c369957b571d74baffe (table default.Table_35d7f864912b40bcb5251a90d40c7849 [id=772d994bee2d4e34bd48912acecf8d48]). Expected version 1 got 0
I20250111 02:57:18.238708 19176 tablet_service.cc:1514] Processing DeleteTablet for tablet b715e32011114e6286c0b8e5406258c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8414653def014ce98bc38d87aac75bae) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:18.346153 19473 raft_consensus.cc:491] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:18.346689 19473 raft_consensus.cc:513] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.347877 19330 tablet_bootstrap.cc:492] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:18.348358 19330 ts_tablet_manager.cc:1397] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.174s user 0.014s sys 0.002s
I20250111 02:57:18.348829 19473 leader_election.cc:290] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:18.350832 19444 raft_consensus.cc:695] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:18.351756 19444 consensus_queue.cc:237] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.353102 18630 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet b8e8c72c99ac4c369957b571d74baffe (table default.Table_35d7f864912b40bcb5251a90d40c7849 [id=772d994bee2d4e34bd48912acecf8d48]). Expected version 1 got 0
I20250111 02:57:18.283563 19325 ts_tablet_manager.cc:1918] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250111 02:57:18.359596 19325 log.cc:1198] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/0d6aa50a4dfe4398a82877b6dfcb5c29
I20250111 02:57:18.360458 19325 ts_tablet_manager.cc:1939] T 0d6aa50a4dfe4398a82877b6dfcb5c29 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:18.366540 19322 tablet_bootstrap.cc:654] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:18.371199 18616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.126:45115, user_credentials={real_user=slave}} blocked reactor thread for 115909us
W20250111 02:57:18.375633 18616 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af -> Peer 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115): Couldn't send request to peer 1e336d4ddf044bf5b0b6e4f5af634727. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250111 02:57:18.384835 19195 raft_consensus.cc:2463] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
W20250111 02:57:18.379501 18615 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 (table default.Table_6883f75ce59a4deaa7ca2c46b1b87f9f [id=b2f59358b7c34e0b88ae3d165f2879b0], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 consensus state has no leader
I20250111 02:57:18.401068 18626 master_service.cc:946] ReplaceTablet: received request to replace tablet 616655aa75824fffa46f81c4f95d5159 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:18.406610 19322 tablet_bootstrap.cc:492] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:18.407321 19322 ts_tablet_manager.cc:1397] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.155s user 0.013s sys 0.005s
I20250111 02:57:18.408761 18622 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 8bfad6c08b4b44c1a664cda3b43acbb1
I20250111 02:57:18.231026 19321 ts_tablet_manager.cc:1397] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.317s user 0.009s sys 0.015s
I20250111 02:57:18.412070 18886 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bfad6c08b4b44c1a664cda3b43acbb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:18.415112 19322 raft_consensus.cc:357] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.416291 19322 raft_consensus.cc:383] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:18.416666 19322 raft_consensus.cc:738] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:18.417542 19322 consensus_queue.cc:260] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.402887 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 65) took 1181 ms. Trace:
I20250111 02:57:18.420138 19325 rpcz_store.cc:276] 0111 02:57:17.221755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:17.221898 (+ 143us) service_pool.cc:225] Handling call
0111 02:57:17.648607 (+426709us) ts_tablet_manager.cc:1145] Deleting tablet 0d6aa50a4dfe4398a82877b6dfcb5c29
0111 02:57:17.648648 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:18.180858 (+532210us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:18.283295 (+102437us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:18.402818 (+119523us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":426006}
W20250111 02:57:18.422384 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 84759.7us
W20250111 02:57:18.424078 18989 leader_election.cc:343] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 149157c1847a497e88a3136a9c1983e9
W20250111 02:57:18.425815 18986 leader_election.cc:343] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:18.426816 18986 leader_election.cc:304] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:18.428239 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet f610a4ff1cc34a928f2db73f14834b25 (table default.Table_fcfe588eae964e7b9f5af53b5728e639 [id=f1373922978a4598b888e1863525942e]) successfully deleted
I20250111 02:57:18.428761 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c02166674a140f29e9f655d09532ced" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:18.428853 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5148fa2e6504561bd1b1a19e48993d7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:18.429438 18909 raft_consensus.cc:2463] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:18.429579 18910 raft_consensus.cc:2463] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:18.438349 19322 ts_tablet_manager.cc:1428] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.031s user 0.005s sys 0.002s
I20250111 02:57:18.439327 19322 tablet_bootstrap.cc:492] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:18.449184 18625 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_c4d2c7776f6c4ad880888a23b8774780" } new_table_name: "default.Table_77baec83405f43feaeb465425798707e" modify_external_catalogs: true
I20250111 02:57:18.449148 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_4dd6f581bd9c4feea48d8aaf545dee06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:18.448585 18626 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_eaa40a643c0e43b9a1204616e7403efa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:18.450753 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dd6f581bd9c4feea48d8aaf545dee06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:18.450778 18886 tablet_service.cc:1514] Processing DeleteTablet for tablet adf93d013fb84ada9ff86ad964273e00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:18.451630 18623 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_b9769d513090402bb596c949defad0df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:18.453709 18623 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9769d513090402bb596c949defad0df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:18.454562 19325 tablet_replica.cc:331] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:18.432188 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c02166674a140f29e9f655d09532ced" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:18.457180 19330 raft_consensus.cc:357] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.457903 19330 raft_consensus.cc:383] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:18.458181 19330 raft_consensus.cc:738] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:18.458755 19330 consensus_queue.cc:260] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.461269 19330 ts_tablet_manager.cc:1428] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.113s user 0.007s sys 0.000s
I20250111 02:57:18.461410 19427 raft_consensus.cc:491] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:18.462249 19330 tablet_bootstrap.cc:492] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:18.461874 19427 raft_consensus.cc:513] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.463883 19427 leader_election.cc:290] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:18.464309 18989 leader_election.cc:304] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:18.464658 19024 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bfad6c08b4b44c1a664cda3b43acbb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:18.415804 19176 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bfad6c08b4b44c1a664cda3b43acbb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:18.465672 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]) successfully deleted
I20250111 02:57:18.466317 19451 raft_consensus.cc:2798] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
W20250111 02:57:18.466444 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet f610a4ff1cc34a928f2db73f14834b25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f610a4ff1cc34a928f2db73f14834b25 already in progress: creating tablet
I20250111 02:57:18.466729 19451 raft_consensus.cc:695] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
W20250111 02:57:18.451014 18626 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eaa40a643c0e43b9a1204616e7403efa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:18.467563 19451 consensus_queue.cc:237] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.468220 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf93d013fb84ada9ff86ad964273e00" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:18.470264 18989 leader_election.cc:304] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters: d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:18.472431 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf93d013fb84ada9ff86ad964273e00" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:18.473234 19053 raft_consensus.cc:2388] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate d0cedab82b154d62b78809edc6f28bfa in this term.
I20250111 02:57:18.476404 18842 leader_election.cc:304] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:18.478435 19427 raft_consensus.cc:3054] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:18.484589 19327 ts_tablet_manager.cc:1918] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:18.485152 19327 log.cc:1198] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/3fa8530e894c498cbbfa73a3ba1c751e
I20250111 02:57:18.486040 19327 ts_tablet_manager.cc:1939] T 3fa8530e894c498cbbfa73a3ba1c751e P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:18.489836 19327 tablet_replica.cc:331] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:18.490556 19327 raft_consensus.cc:2238] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:18.491861 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 3fa8530e894c498cbbfa73a3ba1c751e (table default.Table_c4d2c7776f6c4ad880888a23b8774780 [id=a37b721314134c86b39d1f80b7359f19]) successfully deleted
I20250111 02:57:18.492183 18621 master_service.cc:946] ReplaceTablet: received request to replace tablet 94b9d14959df49a6bbd80ba9e9e57ee4 from {username='slave'} at 127.0.0.1:47522
W20250111 02:57:18.492678 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet c28958a1018344a48d18e6c0f03cd0a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c28958a1018344a48d18e6c0f03cd0a3 already in progress: creating tablet
I20250111 02:57:18.491024 19327 pending_rounds.cc:62] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c: Trying to abort 1 pending ops.
I20250111 02:57:18.495675 19327 pending_rounds.cc:69] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112967114370715648 op_type: NO_OP noop_request { }
I20250111 02:57:18.502765 19474 raft_consensus.cc:2798] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:18.503245 19474 raft_consensus.cc:491] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:18.503590 19474 raft_consensus.cc:3054] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:18.503589 19327 raft_consensus.cc:2883] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:57:18.516060 19400 raft_consensus.cc:2798] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:18.516741 19400 raft_consensus.cc:491] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:18.517190 19400 raft_consensus.cc:3054] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:18.524053 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 97364.2us
I20250111 02:57:18.411507 19321 raft_consensus.cc:357] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.526494 19321 raft_consensus.cc:383] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:18.526809 19321 raft_consensus.cc:738] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
W20250111 02:57:18.527422 18986 leader_election.cc:343] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:18.535487 18986 consensus_peers.cc:487] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:18.545460 19321 consensus_queue.cc:260] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.504069 19327 raft_consensus.cc:2267] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:18.493922 19322 tablet_bootstrap.cc:654] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:18.550393 19325 raft_consensus.cc:2238] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:18.567174 19325 raft_consensus.cc:2267] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:18.569057 19325 ts_tablet_manager.cc:1905] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:18.568274 18614 catalog_manager.cc:4945] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): alter failed for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]),no further retry: Not found: Tablet not found: 0d6aa50a4dfe4398a82877b6dfcb5c29
I20250111 02:57:18.572739 19327 ts_tablet_manager.cc:1905] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:18.573495 19321 ts_tablet_manager.cc:1428] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.164s user 0.005s sys 0.000s
I20250111 02:57:18.574474 19321 tablet_bootstrap.cc:492] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:18.591771 19401 raft_consensus.cc:513] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:18.597366 19133 leader_election.cc:343] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: b715e32011114e6286c0b8e5406258c4
I20250111 02:57:18.598685 19401 leader_election.cc:290] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:18.599874 19427 raft_consensus.cc:2743] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:18.608340 18626 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_346b3997f77e4cfdb856142115ee0efc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:18.577397 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 48806.9us
W20250111 02:57:18.621798 18626 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_346b3997f77e4cfdb856142115ee0efc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:18.639617 19330 tablet_bootstrap.cc:654] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:18.634654 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b715e32011114e6286c0b8e5406258c4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:18.645826 19053 raft_consensus.cc:3054] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:18.612560 19474 raft_consensus.cc:513] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.662196 19474 leader_election.cc:290] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:18.664175 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c02166674a140f29e9f655d09532ced" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:18.665607 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c02166674a140f29e9f655d09532ced" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:18.666340 18910 raft_consensus.cc:3054] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:18.675734 18986 leader_election.cc:343] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:18.682852 19330 tablet_bootstrap.cc:492] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:18.683326 19330 ts_tablet_manager.cc:1397] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.221s user 0.007s sys 0.007s
I20250111 02:57:18.683414 19486 raft_consensus.cc:491] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:18.683880 19486 raft_consensus.cc:513] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.685063 18910 raft_consensus.cc:2463] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:18.685943 19330 raft_consensus.cc:357] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.686602 19330 raft_consensus.cc:383] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:18.686853 19330 raft_consensus.cc:738] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:18.686463 18989 leader_election.cc:304] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters: d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:18.687469 19330 consensus_queue.cc:260] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.689011 19330 ts_tablet_manager.cc:1428] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250111 02:57:18.690876 19330 tablet_bootstrap.cc:492] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:18.690210 19400 raft_consensus.cc:513] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.699766 19330 tablet_bootstrap.cc:654] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:18.699714 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5148fa2e6504561bd1b1a19e48993d7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:18.700425 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066c87476a854e169fd4f5f1cdbb49f7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:18.701227 18986 leader_election.cc:343] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:18.701695 19486 leader_election.cc:290] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
W20250111 02:57:18.702288 18986 leader_election.cc:343] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:18.702575 19486 raft_consensus.cc:2798] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:18.703169 19486 raft_consensus.cc:695] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:18.707242 19400 leader_election.cc:290] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:18.718184 18623 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_064d4893c74f4870bc45572dd481c63c" } modify_external_catalogs: true
I20250111 02:57:18.718803 18623 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_064d4893c74f4870bc45572dd481c63c" } modify_external_catalogs: true
I20250111 02:57:18.682307 19451 consensus_queue.cc:1035] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20250111 02:57:18.727934 19196 raft_consensus.cc:3054] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:18.733402 19486 consensus_queue.cc:237] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:18.736279 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 49935.7us
I20250111 02:57:18.738004 18622 master_service.cc:946] ReplaceTablet: received request to replace tablet 92aa64b2de5746d1b12cc93a10c9d56e from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:18.738044 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066c87476a854e169fd4f5f1cdbb49f7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:18.740324 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5148fa2e6504561bd1b1a19e48993d7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:18.741566 18910 raft_consensus.cc:3054] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:18.744642 19330 tablet_bootstrap.cc:492] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:18.745059 19330 ts_tablet_manager.cc:1397] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.054s user 0.017s sys 0.004s
I20250111 02:57:18.748167 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_293ee09817814cafa48fc9667e47f262"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:18.750550 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_293ee09817814cafa48fc9667e47f262 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:18.770243 19330 raft_consensus.cc:357] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.770929 19330 raft_consensus.cc:383] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:18.771205 19330 raft_consensus.cc:738] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:18.772433 19322 tablet_bootstrap.cc:492] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:18.772859 19322 ts_tablet_manager.cc:1397] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.334s user 0.056s sys 0.050s
I20250111 02:57:18.780447 18626 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_8118351b275b46018f91df033a1270db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:18.783114 18626 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8118351b275b46018f91df033a1270db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:18.788859 19327 ts_tablet_manager.cc:1918] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:18.789503 19327 log.cc:1198] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/6a2234755d474740a55184660119eb11
I20250111 02:57:18.789537 18910 raft_consensus.cc:2463] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:18.790684 19327 ts_tablet_manager.cc:1939] T 6a2234755d474740a55184660119eb11 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:18.794770 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 6a2234755d474740a55184660119eb11 (table default.Table_f05e5d6100bc46379e8598ce5d367e03 [id=9493f23c2c874bda94417f245e37e24f]) successfully deleted
W20250111 02:57:18.797588 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet ef70b417568049f0b496f4d8e5f7e0e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef70b417568049f0b496f4d8e5f7e0e4 already in progress: creating tablet
I20250111 02:57:18.799710 19321 tablet_bootstrap.cc:654] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:18.800148 19327 tablet_replica.cc:331] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:18.813242 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 5a4bc8e9c0c947f6861a01d3e3bd7f31 (DEFAULT_TABLE table=default.Table_a68194fdb909449f878695d09b81c753 [id=49aa9df32f8e4001b66d751832a64534]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:18.832598 19321 tablet_bootstrap.cc:492] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:18.833039 19321 ts_tablet_manager.cc:1397] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.259s user 0.006s sys 0.004s
I20250111 02:57:18.835489 19321 raft_consensus.cc:357] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.836196 19321 raft_consensus.cc:383] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:18.836441 19321 raft_consensus.cc:738] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:18.837132 19321 consensus_queue.cc:260] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.847815 19321 ts_tablet_manager.cc:1428] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.014s user 0.003s sys 0.004s
I20250111 02:57:18.814896 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a4bc8e9c0c947f6861a01d3e3bd7f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:18.852064 19401 raft_consensus.cc:491] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:18.852562 19401 raft_consensus.cc:513] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.852793 18886 tablet_service.cc:1467] Processing CreateTablet for tablet 5a4bc8e9c0c947f6861a01d3e3bd7f31 (DEFAULT_TABLE table=default.Table_a68194fdb909449f878695d09b81c753 [id=49aa9df32f8e4001b66d751832a64534]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:18.854514 18886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a4bc8e9c0c947f6861a01d3e3bd7f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:18.854843 19401 leader_election.cc:290] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:18.856858 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5baef8688ede42fca28cfb6b0bdeefa6" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:18.857721 18910 raft_consensus.cc:2463] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:18.834986 18629 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet b5148fa2e6504561bd1b1a19e48993d7 (table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0]). Expected version 1 got 0
I20250111 02:57:18.859298 19133 leader_election.cc:304] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:18.859866 18629 catalog_manager.cc:5526] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } }
I20250111 02:57:18.861301 18629 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet 5baef8688ede42fca28cfb6b0bdeefa6 (table default.Table_7f4578da9f5e418c9d67a535471b45cd [id=154c37dc0dba44bdab5e0315756dc7cf]). Expected version 1 got 0
I20250111 02:57:18.861423 19401 raft_consensus.cc:2798] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 02:57:18.861801 19401 raft_consensus.cc:491] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:18.862100 19401 raft_consensus.cc:3054] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Advancing to term 2
W20250111 02:57:18.863191 18625 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a983a670c2104398ac1a159ac3a1da7c consensus state has no leader
I20250111 02:57:18.864566 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_91137e45d2b549c6a6c6a478c43c1033"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:18.866180 19321 tablet_bootstrap.cc:492] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
W20250111 02:57:18.866245 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91137e45d2b549c6a6c6a478c43c1033 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:18.871915 19321 tablet_bootstrap.cc:654] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:18.872814 18628 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1802e7349934fbb9d5f7eda37c3df20 consensus state has no leader
I20250111 02:57:18.881469 19169 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6de202f4e547a0b8b90cfa09ad319a (DEFAULT_TABLE table=default.Table_6823aef149894f49822fe303c036446d [id=d5ffe760744748a98f80f25b8092f9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:18.892485 18882 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6de202f4e547a0b8b90cfa09ad319a (DEFAULT_TABLE table=default.Table_6823aef149894f49822fe303c036446d [id=d5ffe760744748a98f80f25b8092f9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:18.893838 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d6de202f4e547a0b8b90cfa09ad319a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:18.895371 19321 tablet_bootstrap.cc:492] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:18.895933 19321 ts_tablet_manager.cc:1397] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.003s
I20250111 02:57:18.895421 19401 raft_consensus.cc:513] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.898337 19321 raft_consensus.cc:357] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.899003 19321 raft_consensus.cc:383] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:18.899294 19321 raft_consensus.cc:738] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:18.900027 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5baef8688ede42fca28cfb6b0bdeefa6" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:18.900009 19321 consensus_queue.cc:260] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.900620 18910 raft_consensus.cc:3054] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:57:18.903358 19401 leader_election.cc:290] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 2 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:18.906211 19321 ts_tablet_manager.cc:1428] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250111 02:57:18.911798 19321 tablet_bootstrap.cc:492] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:18.913319 19427 raft_consensus.cc:491] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:18.913789 19427 raft_consensus.cc:513] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.907608 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d6de202f4e547a0b8b90cfa09ad319a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:18.919906 19427 leader_election.cc:290] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:18.922264 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e8c72c99ac4c369957b571d74baffe" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:18.924574 19321 tablet_bootstrap.cc:654] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:18.927400 18841 leader_election.cc:343] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:18.931241 19133 consensus_peers.cc:487] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fa8530e894c498cbbfa73a3ba1c751e. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:18.935213 19327 raft_consensus.cc:2238] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:18.939244 19174 tablet_service.cc:1467] Processing CreateTablet for tablet cc7158a7fc624fa4bc816ac3721c610d (DEFAULT_TABLE table=default.Table_5f09ae109ece4cd780bf64aa3d7b35c4 [id=94bfb73969874d449273803ecdb4a605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:18.940380 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc7158a7fc624fa4bc816ac3721c610d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:18.955837 19327 raft_consensus.cc:2267] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:18.958070 18627 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 2cd364f9a6894d19826deb59a8d8a612
I20250111 02:57:18.940443 18879 tablet_service.cc:1467] Processing CreateTablet for tablet cc7158a7fc624fa4bc816ac3721c610d (DEFAULT_TABLE table=default.Table_5f09ae109ece4cd780bf64aa3d7b35c4 [id=94bfb73969874d449273803ecdb4a605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:18.960754 18879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc7158a7fc624fa4bc816ac3721c610d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:18.801777 19322 raft_consensus.cc:357] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.961607 19167 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cd364f9a6894d19826deb59a8d8a612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:18.962002 19322 raft_consensus.cc:383] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:18.962286 19322 raft_consensus.cc:738] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:18.962899 19322 consensus_queue.cc:260] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:18.963624 18623 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 9fe4e60db4244a9a9696153b630c575b
I20250111 02:57:18.966971 19167 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe4e60db4244a9a9696153b630c575b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:18.968688 19330 consensus_queue.cc:260] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:18.969879 18989 leader_election.cc:343] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:18.970067 18621 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 94b9d14959df49a6bbd80ba9e9e57ee4
I20250111 02:57:18.970285 18989 leader_election.cc:304] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f; no voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:18.972882 19031 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe4e60db4244a9a9696153b630c575b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:18.973898 19031 tablet_service.cc:1514] Processing DeleteTablet for tablet 94b9d14959df49a6bbd80ba9e9e57ee4 with delete_type TABLET_DATA_DELETED (replaced by tablet e4bd42ad66064bcaa043746081f1c759) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:18.975000 19167 tablet_service.cc:1514] Processing DeleteTablet for tablet 94b9d14959df49a6bbd80ba9e9e57ee4 with delete_type TABLET_DATA_DELETED (replaced by tablet e4bd42ad66064bcaa043746081f1c759) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:18.976748 19327 ts_tablet_manager.cc:1905] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:18.978770 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cd364f9a6894d19826deb59a8d8a612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:18.972110 18621 catalog_manager.cc:6596] ReplaceTablet: tablet 94b9d14959df49a6bbd80ba9e9e57ee4 deleted and replaced by tablet e4bd42ad66064bcaa043746081f1c759
I20250111 02:57:18.980343 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e8c72c99ac4c369957b571d74baffe" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:18.981259 19052 raft_consensus.cc:2463] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:18.983245 19053 raft_consensus.cc:2463] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:18.985198 19023 tablet_service.cc:1467] Processing CreateTablet for tablet 5a4bc8e9c0c947f6861a01d3e3bd7f31 (DEFAULT_TABLE table=default.Table_a68194fdb909449f878695d09b81c753 [id=49aa9df32f8e4001b66d751832a64534]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:18.986740 19023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a4bc8e9c0c947f6861a01d3e3bd7f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:18.987422 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet 94b9d14959df49a6bbd80ba9e9e57ee4 with delete_type TABLET_DATA_DELETED (replaced by tablet e4bd42ad66064bcaa043746081f1c759) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:18.988054 19024 tablet_service.cc:1467] Processing CreateTablet for tablet cc7158a7fc624fa4bc816ac3721c610d (DEFAULT_TABLE table=default.Table_5f09ae109ece4cd780bf64aa3d7b35c4 [id=94bfb73969874d449273803ecdb4a605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:18.988490 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe4e60db4244a9a9696153b630c575b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:18.989385 19322 ts_tablet_manager.cc:1428] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.216s user 0.006s sys 0.002s
I20250111 02:57:18.989624 18910 raft_consensus.cc:2463] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 2.
I20250111 02:57:18.991468 19133 leader_election.cc:304] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:18.992413 19401 raft_consensus.cc:2798] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:57:18.994925 19401 raft_consensus.cc:695] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [term 2 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:18.995554 18909 raft_consensus.cc:1235] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Rejecting Update request from peer 54628fbcec2f4359aa9e26a734d4d29f for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250111 02:57:18.995805 19401 consensus_queue.cc:237] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.000854 19330 ts_tablet_manager.cc:1428] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.255s user 0.009s sys 0.006s
I20250111 02:57:19.000517 19322 tablet_bootstrap.cc:492] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:19.002936 19325 ts_tablet_manager.cc:1918] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:19.003502 19325 log.cc:1198] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/4feb96efdfc44fe4a7cd46dc8deb4fe7
I20250111 02:57:19.005785 19167 tablet_service.cc:1467] Processing CreateTablet for tablet 32afc2f158254811836fcf00f7f409ed (DEFAULT_TABLE table=default.Table_2e1ef9ed4749430f848ae883c239e519 [id=a3db149311a04b458696fd724767db06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.007817 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32afc2f158254811836fcf00f7f409ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.007616 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 32afc2f158254811836fcf00f7f409ed (DEFAULT_TABLE table=default.Table_2e1ef9ed4749430f848ae883c239e519 [id=a3db149311a04b458696fd724767db06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.008864 18878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32afc2f158254811836fcf00f7f409ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.012866 19325 ts_tablet_manager.cc:1939] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:19.024896 19330 tablet_bootstrap.cc:492] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:19.027216 19488 raft_consensus.cc:2743] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:19.031728 19131 leader_election.cc:304] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:19.032903 19444 raft_consensus.cc:2798] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:19.035521 19481 raft_consensus.cc:491] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:19.050982 19321 tablet_bootstrap.cc:492] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:19.052129 19444 raft_consensus.cc:695] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:19.036226 19481 raft_consensus.cc:513] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.053397 19444 consensus_queue.cc:237] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:19.057178 19480 consensus_queue.cc:1035] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
I20250111 02:57:19.053035 19321 ts_tablet_manager.cc:1397] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.141s user 0.012s sys 0.018s
I20250111 02:57:19.063328 19321 raft_consensus.cc:357] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.064043 19321 raft_consensus.cc:383] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:19.064298 19321 raft_consensus.cc:738] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:19.065642 19024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc7158a7fc624fa4bc816ac3721c610d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.065037 19321 consensus_queue.cc:260] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.068259 19321 ts_tablet_manager.cc:1428] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250111 02:57:19.069903 19321 tablet_bootstrap.cc:492] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:19.069348 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cd364f9a6894d19826deb59a8d8a612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:19.021503 19030 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6de202f4e547a0b8b90cfa09ad319a (DEFAULT_TABLE table=default.Table_6823aef149894f49822fe303c036446d [id=d5ffe760744748a98f80f25b8092f9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.075035 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5baef8688ede42fca28cfb6b0bdeefa6" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:18.971503 18989 leader_election.cc:304] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters: d0cedab82b154d62b78809edc6f28bfa
W20250111 02:57:19.076292 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 104913us
I20250111 02:57:19.076682 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5baef8688ede42fca28cfb6b0bdeefa6" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
W20250111 02:57:19.077540 18989 consensus_peers.cc:487] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4feb96efdfc44fe4a7cd46dc8deb4fe7. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:19.078425 19451 raft_consensus.cc:2798] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:19.078526 19480 consensus_queue.cc:1046] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:19.078874 19451 raft_consensus.cc:695] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:19.075277 19030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d6de202f4e547a0b8b90cfa09ad319a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.080166 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e8c72c99ac4c369957b571d74baffe" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:19.083683 19480 raft_consensus.cc:3049] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:57:19.085234 19480 raft_consensus.cc:738] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:19.091758 19321 tablet_bootstrap.cc:654] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:19.094687 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 71) took 1378 ms. Trace:
I20250111 02:57:19.095101 19325 rpcz_store.cc:276] 0111 02:57:17.716662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:17.716780 (+ 118us) service_pool.cc:225] Handling call
0111 02:57:18.454408 (+737628us) ts_tablet_manager.cc:1145] Deleting tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7
0111 02:57:18.454446 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:18.897288 (+442842us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:19.002701 (+105413us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:19.094617 (+ 91916us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16097,"tablet-delete.queue_time_us":736965}
I20250111 02:57:19.096426 19325 tablet_replica.cc:331] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:19.097247 19325 raft_consensus.cc:2238] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:19.097932 19325 pending_rounds.cc:62] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa: Trying to abort 1 pending ops.
I20250111 02:57:19.098145 19325 pending_rounds.cc:69] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112967120328011776 op_type: NO_OP noop_request { }
I20250111 02:57:19.098498 19325 raft_consensus.cc:2883] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:57:19.098860 19325 raft_consensus.cc:2267] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:19.101867 19325 ts_tablet_manager.cc:1905] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:19.104091 19326 raft_consensus.cc:2267] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:19.105557 18842 leader_election.cc:304] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters: d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:19.106966 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e8c72c99ac4c369957b571d74baffe" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:19.107620 18909 raft_consensus.cc:2463] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:19.108409 19427 raft_consensus.cc:2798] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:19.108745 19427 raft_consensus.cc:491] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:19.109059 19427 raft_consensus.cc:3054] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:19.055590 19481 leader_election.cc:290] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:19.084662 19451 consensus_queue.cc:237] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:19.111696 18986 leader_election.cc:343] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:19.026772 19031 tablet_service.cc:1467] Processing CreateTablet for tablet 32afc2f158254811836fcf00f7f409ed (DEFAULT_TABLE table=default.Table_2e1ef9ed4749430f848ae883c239e519 [id=a3db149311a04b458696fd724767db06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.120402 19031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32afc2f158254811836fcf00f7f409ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.086753 19480 consensus_queue.cc:260] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.124787 19480 raft_consensus.cc:3054] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:57:19.125658 18989 leader_election.cc:304] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters: d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:19.129767 19481 raft_consensus.cc:2798] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:19.130128 19481 raft_consensus.cc:491] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:19.130443 19481 raft_consensus.cc:3054] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:19.136739 19473 raft_consensus.cc:491] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:19.137543 19473 raft_consensus.cc:513] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.139664 19330 tablet_bootstrap.cc:654] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:19.145084 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa8530e894c498cbbfa73a3ba1c751e" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:19.146550 18614 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 105918us
I20250111 02:57:19.148531 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 (table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0]) successfully deleted
I20250111 02:57:19.147486 19473 leader_election.cc:290] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:19.158654 19195 raft_consensus.cc:1235] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [term 2 LEADER]: Rejecting Update request from peer 54628fbcec2f4359aa9e26a734d4d29f for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250111 02:57:19.167397 18879 tablet_service.cc:1467] Processing CreateTablet for tablet 49616e4e7cc647509e72cde57b87d6fa (DEFAULT_TABLE table=default.Table_c4faffc9659848ea9bff581e8d88d40e [id=e709d8f708144de8a18d241f0e0306a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.170500 18882 tablet_service.cc:1467] Processing CreateTablet for tablet 1b69a8c2caea4bddac3f539733bcd3af (DEFAULT_TABLE table=default.Table_bf50e4c0392544f8a24b861d3edaf46e [id=4b08fa373b0b460094c09b4d3353951c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.172099 18625 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_4672fdb5296042f3b9373401c98fa509" } new_table_name: "default.Table_8292a1a0b58247ca951f8f8a951a83ad" modify_external_catalogs: true
I20250111 02:57:19.173224 18880 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea78feee64340e39404cba0f1bf04db (DEFAULT_TABLE table=default.Table_a4c544f93b2a4ad5afd4aa78f3989b6f [id=cf9df41fb95c4e9c89cf0f52217843a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.175766 18624 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_7ffc93386dff41da8bbdaf9f883bc27b" } new_table_name: "default.Table_8776da962d214139886beb69df33db00" modify_external_catalogs: true
I20250111 02:57:19.176383 19034 tablet_service.cc:1467] Processing CreateTablet for tablet 1b69a8c2caea4bddac3f539733bcd3af (DEFAULT_TABLE table=default.Table_bf50e4c0392544f8a24b861d3edaf46e [id=4b08fa373b0b460094c09b4d3353951c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.176733 19033 tablet_service.cc:1467] Processing CreateTablet for tablet 49616e4e7cc647509e72cde57b87d6fa (DEFAULT_TABLE table=default.Table_c4faffc9659848ea9bff581e8d88d40e [id=e709d8f708144de8a18d241f0e0306a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.177608 19034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b69a8c2caea4bddac3f539733bcd3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.178210 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49616e4e7cc647509e72cde57b87d6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.180414 18623 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_0791d732a14a40bbadc0dadb66c22f0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:19.156386 18989 leader_election.cc:343] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 3fa8530e894c498cbbfa73a3ba1c751e
I20250111 02:57:19.181866 18989 leader_election.cc:304] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f; no voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:19.182720 19473 raft_consensus.cc:2743] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:19.184791 18628 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_2e1ef9ed4749430f848ae883c239e519" } modify_external_catalogs: true
I20250111 02:57:19.185755 18627 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_857f04d0de7e47a4828db9a8997f1009"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:19.188802 18628 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_2e1ef9ed4749430f848ae883c239e519" } modify_external_catalogs: true
W20250111 02:57:19.190865 18623 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0791d732a14a40bbadc0dadb66c22f0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:19.193969 19327 ts_tablet_manager.cc:1918] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:19.194361 19327 log.cc:1198] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/8bfad6c08b4b44c1a664cda3b43acbb1
I20250111 02:57:19.195036 19327 ts_tablet_manager.cc:1939] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
W20250111 02:57:19.204308 18627 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_857f04d0de7e47a4828db9a8997f1009 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:19.225553 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet 616655aa75824fffa46f81c4f95d5159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:19.231853 19401 raft_consensus.cc:491] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:19.232312 19401 raft_consensus.cc:513] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.234573 19401 leader_election.cc:290] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
W20250111 02:57:19.237635 19133 leader_election.cc:343] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: f610a4ff1cc34a928f2db73f14834b25
I20250111 02:57:19.240218 19327 tablet_replica.cc:331] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:19.241003 19327 raft_consensus.cc:2238] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:19.241467 19327 raft_consensus.cc:2267] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:19.243759 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 8bfad6c08b4b44c1a664cda3b43acbb1 (table default.Table_65af1591521e4974a28304ce51fe6fb1 [id=7cf2761276494576a4c2be6239374334]) successfully deleted
I20250111 02:57:19.245950 19327 ts_tablet_manager.cc:1905] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:19.251080 19052 raft_consensus.cc:3054] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:19.257797 19427 raft_consensus.cc:513] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:19.258423 19131 leader_election.cc:343] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: f610a4ff1cc34a928f2db73f14834b25
I20250111 02:57:19.258826 19131 leader_election.cc:304] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:19.259817 19401 raft_consensus.cc:2743] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:19.259884 19427 leader_election.cc:290] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:19.262365 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e8c72c99ac4c369957b571d74baffe" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:19.262667 19322 tablet_bootstrap.cc:654] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:19.264039 18841 leader_election.cc:343] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:19.267453 18879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49616e4e7cc647509e72cde57b87d6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.269133 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b69a8c2caea4bddac3f539733bcd3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.270488 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ea78feee64340e39404cba0f1bf04db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.285658 19167 tablet_service.cc:1467] Processing CreateTablet for tablet 49616e4e7cc647509e72cde57b87d6fa (DEFAULT_TABLE table=default.Table_c4faffc9659848ea9bff581e8d88d40e [id=e709d8f708144de8a18d241f0e0306a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.287197 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 1b69a8c2caea4bddac3f539733bcd3af (DEFAULT_TABLE table=default.Table_bf50e4c0392544f8a24b861d3edaf46e [id=4b08fa373b0b460094c09b4d3353951c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.292953 19325 ts_tablet_manager.cc:1918] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:19.293350 19325 log.cc:1198] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/adf93d013fb84ada9ff86ad964273e00
I20250111 02:57:19.294009 19325 ts_tablet_manager.cc:1939] T adf93d013fb84ada9ff86ad964273e00 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:19.294695 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49616e4e7cc647509e72cde57b87d6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.296005 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b69a8c2caea4bddac3f539733bcd3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.305368 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 72) took 1587 ms. Trace:
I20250111 02:57:19.312824 18881 tablet_service.cc:1467] Processing CreateTablet for tablet 4b302fcba27244c193586bc25fc9305f (DEFAULT_TABLE table=default.Table_4672fdb5296042f3b9373401c98fa509 [id=d7462be28a2c445889cf333c56ae28a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.314044 18881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b302fcba27244c193586bc25fc9305f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.313539 19325 rpcz_store.cc:276] 0111 02:57:17.717970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:17.718069 (+ 99us) service_pool.cc:225] Handling call
0111 02:57:19.096254 (+1378185us) ts_tablet_manager.cc:1145] Deleting tablet adf93d013fb84ada9ff86ad964273e00
0111 02:57:19.096303 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:19.218939 (+122636us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:19.292708 (+ 73769us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:19.305303 (+ 12595us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1377587}
I20250111 02:57:19.315488 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 78) took 1249 ms. Trace:
I20250111 02:57:19.315922 19325 rpcz_store.cc:276] 0111 02:57:18.065786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.065945 (+ 159us) service_pool.cc:225] Handling call
0111 02:57:19.315110 (+1249165us) ts_tablet_manager.cc:1145] Deleting tablet c28958a1018344a48d18e6c0f03cd0a3
0111 02:57:19.315160 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:19.315429 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1248486}
I20250111 02:57:19.317477 19325 tablet_replica.cc:331] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:19.318324 19325 raft_consensus.cc:2238] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:19.319789 18630 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 5baef8688ede42fca28cfb6b0bdeefa6 (table default.Table_7f4578da9f5e418c9d67a535471b45cd [id=154c37dc0dba44bdab5e0315756dc7cf]). Expected version 1 got 0
W20250111 02:57:19.321501 18630 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.67:53171 (request call id 24) took 756 ms (client timeout 1000 ms). Trace:
W20250111 02:57:19.321852 18630 rpcz_store.cc:269] 0111 02:57:18.564874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.565800 (+ 926us) service_pool.cc:225] Handling call
0111 02:57:19.321427 (+755627us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20250111 02:57:19.323616 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet adf93d013fb84ada9ff86ad964273e00 (table default.Table_baed23f14d1d4ebb978e45731f534aba [id=ec872ce86d1940cb85adae5f3047321d]) successfully deleted
W20250111 02:57:19.324831 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet c28958a1018344a48d18e6c0f03cd0a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c28958a1018344a48d18e6c0f03cd0a3 already in progress: creating tablet
I20250111 02:57:19.324750 19169 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea78feee64340e39404cba0f1bf04db (DEFAULT_TABLE table=default.Table_a4c544f93b2a4ad5afd4aa78f3989b6f [id=cf9df41fb95c4e9c89cf0f52217843a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.326287 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ea78feee64340e39404cba0f1bf04db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.323415 19325 raft_consensus.cc:2267] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:19.338022 19326 ts_tablet_manager.cc:1905] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:19.340077 19325 ts_tablet_manager.cc:1905] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:19.342504 18629 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.66:37049 (request call id 26) took 1008 ms (client timeout 1000 ms). Trace:
I20250111 02:57:19.350224 18622 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 92aa64b2de5746d1b12cc93a10c9d56e
I20250111 02:57:19.356629 18622 catalog_manager.cc:6596] ReplaceTablet: tablet 92aa64b2de5746d1b12cc93a10c9d56e deleted and replaced by tablet e46894710a72407993b36801f3b90eb0
I20250111 02:57:19.360388 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet 92aa64b2de5746d1b12cc93a10c9d56e with delete_type TABLET_DATA_DELETED (replaced by tablet e46894710a72407993b36801f3b90eb0) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:19.365638 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet 92aa64b2de5746d1b12cc93a10c9d56e with delete_type TABLET_DATA_DELETED (replaced by tablet e46894710a72407993b36801f3b90eb0) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:19.369836 19174 tablet_service.cc:1467] Processing CreateTablet for tablet 4b302fcba27244c193586bc25fc9305f (DEFAULT_TABLE table=default.Table_4672fdb5296042f3b9373401c98fa509 [id=d7462be28a2c445889cf333c56ae28a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.371162 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b302fcba27244c193586bc25fc9305f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.371106 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e8c72c99ac4c369957b571d74baffe" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:19.336338 19481 raft_consensus.cc:513] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.375406 19026 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea78feee64340e39404cba0f1bf04db (DEFAULT_TABLE table=default.Table_a4c544f93b2a4ad5afd4aa78f3989b6f [id=cf9df41fb95c4e9c89cf0f52217843a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.374256 18882 tablet_service.cc:1467] Processing CreateTablet for tablet f778763e59a94969b3f62be87e2fd867 (DEFAULT_TABLE table=default.Table_902d917b19634ad5b20fadd455c0ba88 [id=ec311d46e9224fd38d1864b43717da4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.377002 19481 leader_election.cc:290] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:19.378911 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e8c72c99ac4c369957b571d74baffe" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:19.380511 18909 raft_consensus.cc:2388] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 54628fbcec2f4359aa9e26a734d4d29f in current term 1: Already voted for candidate 8221604733fe4fc0a1fc32de8d6af71c in this term.
I20250111 02:57:19.380571 19170 tablet_service.cc:1467] Processing CreateTablet for tablet f778763e59a94969b3f62be87e2fd867 (DEFAULT_TABLE table=default.Table_902d917b19634ad5b20fadd455c0ba88 [id=ec311d46e9224fd38d1864b43717da4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.381742 19170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f778763e59a94969b3f62be87e2fd867. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.382947 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 845da3ee0863430ea3fe9de8c84f2c17 (DEFAULT_TABLE table=default.Table_292c103f235f421db8dacf834cca2c53 [id=b091f6cd63104693abbdd739bca159c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.383436 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e8c72c99ac4c369957b571d74baffe" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
W20250111 02:57:19.384953 18986 leader_election.cc:343] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:19.385413 18986 leader_election.cc:304] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f; no voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:19.386631 19481 raft_consensus.cc:2743] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250111 02:57:19.392442 19032 tablet_service.cc:1514] Processing DeleteTablet for tablet adf93d013fb84ada9ff86ad964273e00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:19.393537 19025 tablet_service.cc:1514] Processing DeleteTablet for tablet 72bcbe140a82462d909bd6d41bfbd2af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:14 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:19.394273 19028 tablet_service.cc:1514] Processing DeleteTablet for tablet b715e32011114e6286c0b8e5406258c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8414653def014ce98bc38d87aac75bae) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:19.395006 19027 tablet_service.cc:1514] Processing DeleteTablet for tablet 149157c1847a497e88a3136a9c1983e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:19.395627 19029 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa8530e894c498cbbfa73a3ba1c751e with delete_type TABLET_DATA_DELETED (replaced by tablet a983a670c2104398ac1a159ac3a1da7c) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:19.396569 19027 tablet_service.cc:1514] Processing DeleteTablet for tablet f610a4ff1cc34a928f2db73f14834b25 with delete_type TABLET_DATA_DELETED (replaced by tablet 855dac47872d4c5fbf8f5e38b588c7fe) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:19.398537 19054 raft_consensus.cc:2388] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:19.401535 18842 leader_election.cc:304] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:19.403645 19026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ea78feee64340e39404cba0f1bf04db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.407846 19027 tablet_service.cc:1514] Processing DeleteTablet for tablet 92aa64b2de5746d1b12cc93a10c9d56e with delete_type TABLET_DATA_DELETED (replaced by tablet e46894710a72407993b36801f3b90eb0) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:19.410441 19030 tablet_service.cc:1467] Processing CreateTablet for tablet f778763e59a94969b3f62be87e2fd867 (DEFAULT_TABLE table=default.Table_902d917b19634ad5b20fadd455c0ba88 [id=ec311d46e9224fd38d1864b43717da4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.411932 19030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f778763e59a94969b3f62be87e2fd867. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.412889 18879 tablet_service.cc:1467] Processing CreateTablet for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa (DEFAULT_TABLE table=default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.415222 19027 tablet_service.cc:1467] Processing CreateTablet for tablet 4b302fcba27244c193586bc25fc9305f (DEFAULT_TABLE table=default.Table_4672fdb5296042f3b9373401c98fa509 [id=d7462be28a2c445889cf333c56ae28a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.416376 19027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b302fcba27244c193586bc25fc9305f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.420200 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 845da3ee0863430ea3fe9de8c84f2c17 (DEFAULT_TABLE table=default.Table_292c103f235f421db8dacf834cca2c53 [id=b091f6cd63104693abbdd739bca159c3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250111 02:57:19.424702 18615 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet a983a670c2104398ac1a159ac3a1da7c (table default.Table_77baec83405f43feaeb465425798707e [id=a37b721314134c86b39d1f80b7359f19], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet a983a670c2104398ac1a159ac3a1da7c consensus state has no leader
I20250111 02:57:19.425187 19401 raft_consensus.cc:491] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:19.425683 19401 raft_consensus.cc:513] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.427750 19401 leader_election.cc:290] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:19.427804 19029 tablet_service.cc:1467] Processing CreateTablet for tablet 845da3ee0863430ea3fe9de8c84f2c17 (DEFAULT_TABLE table=default.Table_292c103f235f421db8dacf834cca2c53 [id=b091f6cd63104693abbdd739bca159c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.429946 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5148fa2e6504561bd1b1a19e48993d7" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:19.430788 18909 raft_consensus.cc:2388] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0cedab82b154d62b78809edc6f28bfa in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:19.431226 19023 tablet_service.cc:1467] Processing CreateTablet for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa (DEFAULT_TABLE table=default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.431839 18880 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf1ce65ccf54358b95665b9764fbdb0 (DEFAULT_TABLE table=default.Table_61c40191c611402da3829f2ef5379f61 [id=b4790f95b19748c7a1130504d33b0319]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.433579 19167 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf1ce65ccf54358b95665b9764fbdb0 (DEFAULT_TABLE table=default.Table_61c40191c611402da3829f2ef5379f61 [id=b4790f95b19748c7a1130504d33b0319]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.434435 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5148fa2e6504561bd1b1a19e48993d7" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:19.436105 19131 leader_election.cc:304] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:19.437345 19401 raft_consensus.cc:3054] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:19.437271 19427 raft_consensus.cc:2743] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250111 02:57:19.438493 19028 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf1ce65ccf54358b95665b9764fbdb0 (DEFAULT_TABLE table=default.Table_61c40191c611402da3829f2ef5379f61 [id=b4790f95b19748c7a1130504d33b0319]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.438573 19169 tablet_service.cc:1467] Processing CreateTablet for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa (DEFAULT_TABLE table=default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]), partition=RANGE (key) PARTITION UNBOUNDED
W20250111 02:57:19.343448 18629 rpcz_store.cc:269] 0111 02:57:18.334362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.334502 (+ 140us) service_pool.cc:225] Handling call
0111 02:57:19.342427 (+1007925us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:18.997608 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:19.017694 (+ 20086us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:19.017731 (+ 37us) write_op.cc:435] Acquired schema lock
0111 02:57:19.017752 (+ 21us) tablet.cc:659] Decoding operations
0111 02:57:19.018126 (+ 374us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:19.018186 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:19.018220 (+ 34us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:19.018368 (+ 148us) tablet.cc:698] Row locks acquired
0111 02:57:19.018389 (+ 21us) write_op.cc:260] PREPARE: finished
0111 02:57:19.018544 (+ 155us) write_op.cc:270] Start()
0111 02:57:19.018658 (+ 114us) write_op.cc:276] Timestamp: P: 1736564239018450 usec, L: 0
0111 02:57:19.018680 (+ 22us) op_driver.cc:348] REPLICATION: starting
0111 02:57:19.174147 (+155467us) log.cc:844] Serialized 530 byte log entry
0111 02:57:19.227831 (+ 53684us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f316d1 00007fd427fa8198 00007fd4284143b1 00007fd428413b88 00007fd428418821 00007fd4284187d9 00007fd428418769 00007fd428418731 00007fd42841795d 000000000059ea04 000000000059e839
0111 02:57:19.314744 (+ 86913us) op_driver.cc:464] REPLICATION: finished
0111 02:57:19.315018 (+ 274us) write_op.cc:301] APPLY: starting
0111 02:57:19.315103 (+ 85us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:19.315209 (+ 106us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:19.315227 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:19.315395 (+ 168us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:19.315545 (+ 150us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:19.315587 (+ 42us) write_op.cc:312] APPLY: finished
0111 02:57:19.315782 (+ 195us) log.cc:844] Serialized 23 byte log entry
0111 02:57:19.315939 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:19.316010 (+ 71us) write_op.cc:454] Released schema lock
0111 02:57:19.316041 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":105,"apply.run_cpu_time_us":1255,"apply.run_wall_time_us":1254,"num_ops":1,"prepare.queue_time_us":122263,"prepare.run_cpu_time_us":3873,"prepare.run_wall_time_us":230427,"raft.queue_time_us":3326,"raft.run_cpu_time_us":1236,"raft.run_wall_time_us":29539,"replication_time_us":295943,"spinlock_wait_cycles":355449216}]]}
I20250111 02:57:19.444342 18629 master_service.cc:946] ReplaceTablet: received request to replace tablet ede92343d9b54c29b8173a5eafcf68c6 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:19.445354 18878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 845da3ee0863430ea3fe9de8c84f2c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.445415 19327 ts_tablet_manager.cc:1918] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:19.445842 19327 log.cc:1198] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/adf93d013fb84ada9ff86ad964273e00
I20250111 02:57:19.446041 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f778763e59a94969b3f62be87e2fd867. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.446182 18884 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd364f9a6894d19826deb59a8d8a612 (DEFAULT_TABLE table=default.Table_71ceb23d653647089e7bd2fa3da23c01 [id=7e3413aa9c8d4037928c497a8afad14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.446523 19327 ts_tablet_manager.cc:1939] T adf93d013fb84ada9ff86ad964273e00 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:19.446650 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf1ce65ccf54358b95665b9764fbdb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.447367 18884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd364f9a6894d19826deb59a8d8a612. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.447391 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 845da3ee0863430ea3fe9de8c84f2c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.448638 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.450984 18879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.461184 19330 tablet_bootstrap.cc:492] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
W20250111 02:57:19.462821 18989 consensus_peers.cc:487] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 149157c1847a497e88a3136a9c1983e9. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:19.472571 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf1ce65ccf54358b95665b9764fbdb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.481658 19168 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd364f9a6894d19826deb59a8d8a612 (DEFAULT_TABLE table=default.Table_71ceb23d653647089e7bd2fa3da23c01 [id=7e3413aa9c8d4037928c497a8afad14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.482349 19427 raft_consensus.cc:491] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:19.482790 19427 raft_consensus.cc:513] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.485116 19025 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd364f9a6894d19826deb59a8d8a612 (DEFAULT_TABLE table=default.Table_71ceb23d653647089e7bd2fa3da23c01 [id=7e3413aa9c8d4037928c497a8afad14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.488572 19170 tablet_service.cc:1467] Processing CreateTablet for tablet 0158a8849bf44a4fbb5daca22883478f (DEFAULT_TABLE table=default.Table_c2fa721b96ba4e9ca293ff4b5c455447 [id=64d78652fc6f44778133bd3157964db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.490468 19401 raft_consensus.cc:2743] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:19.492579 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 83) took 1042 ms. Trace:
I20250111 02:57:19.493067 19029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 845da3ee0863430ea3fe9de8c84f2c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.493013 19327 rpcz_store.cc:276] 0111 02:57:18.450374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.450610 (+ 236us) service_pool.cc:225] Handling call
0111 02:57:19.240062 (+789452us) ts_tablet_manager.cc:1145] Deleting tablet adf93d013fb84ada9ff86ad964273e00
0111 02:57:19.240100 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:19.294619 (+ 54519us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:19.445176 (+150557us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:19.492508 (+ 47332us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":788811}
I20250111 02:57:19.494566 19025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd364f9a6894d19826deb59a8d8a612. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.495939 19401 consensus_queue.cc:1035] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:19.496165 19028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf1ce65ccf54358b95665b9764fbdb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.496304 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet adf93d013fb84ada9ff86ad964273e00 (table default.Table_baed23f14d1d4ebb978e45731f534aba [id=ec872ce86d1940cb85adae5f3047321d]) successfully deleted
W20250111 02:57:19.497561 18617 catalog_manager.cc:4839] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 2cd364f9a6894d19826deb59a8d8a612 because the tablet was not found. No further retry: Not found: Tablet not found: 2cd364f9a6894d19826deb59a8d8a612
I20250111 02:57:19.498502 19325 ts_tablet_manager.cc:1918] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:19.498854 19325 log.cc:1198] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/149157c1847a497e88a3136a9c1983e9
I20250111 02:57:19.499578 19325 ts_tablet_manager.cc:1939] T 149157c1847a497e88a3136a9c1983e9 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
W20250111 02:57:19.502125 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 94b9d14959df49a6bbd80ba9e9e57ee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94b9d14959df49a6bbd80ba9e9e57ee4 already in progress: creating tablet
I20250111 02:57:19.502667 19195 raft_consensus.cc:3054] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:19.503424 19327 tablet_replica.cc:331] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:19.504096 19327 raft_consensus.cc:2238] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:19.504537 19327 raft_consensus.cc:2267] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus is shut down!
W20250111 02:57:19.505750 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 9fe4e60db4244a9a9696153b630c575b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fe4e60db4244a9a9696153b630c575b already in progress: creating tablet
I20250111 02:57:19.506778 19168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd364f9a6894d19826deb59a8d8a612. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.507151 19023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.510440 19170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0158a8849bf44a4fbb5daca22883478f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.511933 18886 tablet_service.cc:1467] Processing CreateTablet for tablet 0158a8849bf44a4fbb5daca22883478f (DEFAULT_TABLE table=default.Table_c2fa721b96ba4e9ca293ff4b5c455447 [id=64d78652fc6f44778133bd3157964db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.526140 19024 tablet_service.cc:1467] Processing CreateTablet for tablet 0158a8849bf44a4fbb5daca22883478f (DEFAULT_TABLE table=default.Table_c2fa721b96ba4e9ca293ff4b5c455447 [id=64d78652fc6f44778133bd3157964db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.527474 19024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0158a8849bf44a4fbb5daca22883478f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.529492 18886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0158a8849bf44a4fbb5daca22883478f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.534297 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 80) took 1431 ms. Trace:
I20250111 02:57:19.461652 19330 ts_tablet_manager.cc:1397] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.437s user 0.020s sys 0.010s
I20250111 02:57:19.534677 19325 rpcz_store.cc:276] 0111 02:57:18.103132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.103303 (+ 171us) service_pool.cc:225] Handling call
0111 02:57:19.317284 (+1213981us) ts_tablet_manager.cc:1145] Deleting tablet 149157c1847a497e88a3136a9c1983e9
0111 02:57:19.317339 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:19.449919 (+132580us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:19.495557 (+ 45638us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:19.534232 (+ 38675us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1213340}
I20250111 02:57:19.535889 19325 tablet_replica.cc:331] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:19.535562 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_d37d3e82138442838c482c283e2d1002"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:19.535980 18621 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_c2796c2dda744b6c8c8b1b8a7bed2a19" } modify_external_catalogs: true
I20250111 02:57:19.536598 18621 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_c2796c2dda744b6c8c8b1b8a7bed2a19" } modify_external_catalogs: true
W20250111 02:57:19.537082 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d37d3e82138442838c482c283e2d1002 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:19.539706 18882 tablet_service.cc:1467] Processing CreateTablet for tablet 8414653def014ce98bc38d87aac75bae (DEFAULT_TABLE table=default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.541244 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8414653def014ce98bc38d87aac75bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.541157 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 149157c1847a497e88a3136a9c1983e9 (table default.Table_d76c4a77f7d244478357d5a5fc3cdda6 [id=b46d11174b994b818737139b9b2eb13a]) successfully deleted
I20250111 02:57:19.548075 18890 tablet_service.cc:1467] Processing CreateTablet for tablet 855dac47872d4c5fbf8f5e38b588c7fe (DEFAULT_TABLE table=default.Table_fcfe588eae964e7b9f5af53b5728e639 [id=f1373922978a4598b888e1863525942e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.549261 18890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 855dac47872d4c5fbf8f5e38b588c7fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.552120 19033 tablet_service.cc:1467] Processing CreateTablet for tablet 855dac47872d4c5fbf8f5e38b588c7fe (DEFAULT_TABLE table=default.Table_fcfe588eae964e7b9f5af53b5728e639 [id=f1373922978a4598b888e1863525942e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.553274 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 855dac47872d4c5fbf8f5e38b588c7fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.558374 19174 tablet_service.cc:1467] Processing CreateTablet for tablet 855dac47872d4c5fbf8f5e38b588c7fe (DEFAULT_TABLE table=default.Table_fcfe588eae964e7b9f5af53b5728e639 [id=f1373922978a4598b888e1863525942e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.560010 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 855dac47872d4c5fbf8f5e38b588c7fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.563652 19031 tablet_service.cc:1467] Processing CreateTablet for tablet 8414653def014ce98bc38d87aac75bae (DEFAULT_TABLE table=default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.566402 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 8414653def014ce98bc38d87aac75bae (DEFAULT_TABLE table=default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:19.567515 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8414653def014ce98bc38d87aac75bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.536554 19325 raft_consensus.cc:2238] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:19.571877 19325 pending_rounds.cc:62] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa: Trying to abort 1 pending ops.
I20250111 02:57:19.572165 19325 pending_rounds.cc:69] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112967118953123840 op_type: NO_OP noop_request { }
I20250111 02:57:19.572520 19325 raft_consensus.cc:2883] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:57:19.572822 19325 raft_consensus.cc:2267] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:19.589545 19322 tablet_bootstrap.cc:492] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:19.589987 19322 ts_tablet_manager.cc:1397] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.590s user 0.009s sys 0.002s
I20250111 02:57:19.596427 19325 ts_tablet_manager.cc:1905] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:19.607195 19031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8414653def014ce98bc38d87aac75bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:19.608889 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616655aa75824fffa46f81c4f95d5159" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:19.610417 18841 leader_election.cc:343] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:19.610373 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616655aa75824fffa46f81c4f95d5159" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:19.611871 18842 leader_election.cc:343] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:19.612305 18842 leader_election.cc:304] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
W20250111 02:57:19.613065 18842 raft_consensus.cc:2681] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250111 02:57:19.622789 19321 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.009s sys 0.008s
W20250111 02:57:19.623167 19321 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.507s user 0.009s sys 0.008s
I20250111 02:57:19.628865 19427 leader_election.cc:290] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:19.630730 18624 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8ef3f45e5784a29b3b32cc0de6bac84 consensus state has no leader
I20250111 02:57:19.631356 19327 ts_tablet_manager.cc:1905] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:19.633711 18628 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 32afc2f158254811836fcf00f7f409ed
I20250111 02:57:19.640434 19169 tablet_service.cc:1514] Processing DeleteTablet for tablet 32afc2f158254811836fcf00f7f409ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:19 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:19.658185 19322 raft_consensus.cc:357] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.659179 19322 raft_consensus.cc:383] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:19.662271 18628 master_service.cc:946] ReplaceTablet: received request to replace tablet 5118d4b9cfbb49e8af6af56180858336 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:19.662956 19322 raft_consensus.cc:738] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:19.649638 19321 tablet_bootstrap.cc:492] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:19.667572 19321 ts_tablet_manager.cc:1397] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.598s user 0.017s sys 0.010s
I20250111 02:57:19.674772 19321 raft_consensus.cc:357] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:19.682585 18625 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b302fcba27244c193586bc25fc9305f consensus state has no leader
I20250111 02:57:19.681066 19330 raft_consensus.cc:357] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.685238 19322 consensus_queue.cc:260] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.685664 19330 raft_consensus.cc:383] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:19.686069 19330 raft_consensus.cc:738] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:19.687379 19322 ts_tablet_manager.cc:1428] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.097s user 0.008s sys 0.000s
I20250111 02:57:19.687844 18879 tablet_service.cc:1514] Processing DeleteTablet for tablet 32afc2f158254811836fcf00f7f409ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:19 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:19.689942 19321 raft_consensus.cc:383] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:19.697201 19321 raft_consensus.cc:738] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:19.700212 19321 consensus_queue.cc:260] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.723662 19322 tablet_bootstrap.cc:492] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:19.735536 19321 ts_tablet_manager.cc:1428] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.066s user 0.005s sys 0.001s
I20250111 02:57:19.741128 19325 ts_tablet_manager.cc:1918] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:19.744964 19325 log.cc:1198] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/3fa8530e894c498cbbfa73a3ba1c751e
I20250111 02:57:19.745931 19325 ts_tablet_manager.cc:1939] T 3fa8530e894c498cbbfa73a3ba1c751e P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:19.746762 19321 tablet_bootstrap.cc:492] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:19.762543 19322 tablet_bootstrap.cc:654] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:19.777768 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_69d080492ff2442ea30d95edf1c27459"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:19.784025 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69d080492ff2442ea30d95edf1c27459 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:19.686898 19330 consensus_queue.cc:260] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.793449 19427 raft_consensus.cc:491] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:19.793947 19427 raft_consensus.cc:513] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.796005 19427 leader_election.cc:290] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:19.797206 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede92343d9b54c29b8173a5eafcf68c6" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:19.797891 19195 raft_consensus.cc:2463] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:19.799285 18841 leader_election.cc:304] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:19.800631 19427 raft_consensus.cc:2798] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:19.801048 19427 raft_consensus.cc:491] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:19.801455 19427 raft_consensus.cc:3054] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:19.807088 18623 master_service.cc:946] ReplaceTablet: received request to replace tablet 32afc2f158254811836fcf00f7f409ed from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:19.800177 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede92343d9b54c29b8173a5eafcf68c6" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:19.824050 18842 leader_election.cc:343] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:19.825325 19330 ts_tablet_manager.cc:1428] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.290s user 0.007s sys 0.000s
I20250111 02:57:19.826261 19330 tablet_bootstrap.cc:492] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:19.832211 19401 raft_consensus.cc:491] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:19.766953 19321 tablet_bootstrap.cc:654] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:19.804824 19326 ts_tablet_manager.cc:1918] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:19.832703 19401 raft_consensus.cc:513] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.688650 19451 consensus_queue.cc:1035] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.045s
I20250111 02:57:19.834533 19401 leader_election.cc:290] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:19.836920 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c02166674a140f29e9f655d09532ced" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:19.837622 18909 raft_consensus.cc:2463] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:19.841600 18627 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_77baec83405f43feaeb465425798707e" } new_table_name: "default.Table_e1f7351012894b2e9e3dde1c1006874d" modify_external_catalogs: true
I20250111 02:57:19.844456 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 81) took 1740 ms. Trace:
I20250111 02:57:19.844794 19325 rpcz_store.cc:276] 0111 02:57:18.104327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.104469 (+ 142us) service_pool.cc:225] Handling call
0111 02:57:19.535749 (+1431280us) ts_tablet_manager.cc:1145] Deleting tablet 3fa8530e894c498cbbfa73a3ba1c751e
0111 02:57:19.535787 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:19.652357 (+116570us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:19.740807 (+ 88450us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:19.844391 (+103584us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1430695}
I20250111 02:57:19.845896 19325 tablet_replica.cc:331] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:19.846627 19325 raft_consensus.cc:2238] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:19.847604 19327 ts_tablet_manager.cc:1918] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:19.847550 19133 leader_election.cc:304] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:19.847986 19327 log.cc:1198] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/616655aa75824fffa46f81c4f95d5159
I20250111 02:57:19.848718 19322 tablet_bootstrap.cc:492] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:19.848980 19444 raft_consensus.cc:2798] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 02:57:19.849153 19322 ts_tablet_manager.cc:1397] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.126s user 0.011s sys 0.000s
I20250111 02:57:19.849321 19444 raft_consensus.cc:491] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:19.849634 19444 raft_consensus.cc:3054] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:57:19.849807 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 3fa8530e894c498cbbfa73a3ba1c751e (table default.Table_77baec83405f43feaeb465425798707e [id=a37b721314134c86b39d1f80b7359f19]) successfully deleted
I20250111 02:57:19.848722 19327 ts_tablet_manager.cc:1939] T 616655aa75824fffa46f81c4f95d5159 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
W20250111 02:57:19.819345 18986 consensus_peers.cc:487] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 149157c1847a497e88a3136a9c1983e9. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:19.856038 19322 raft_consensus.cc:357] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.856792 19322 raft_consensus.cc:383] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:19.856824 18624 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_0791d732a14a40bbadc0dadb66c22f0c" } modify_external_catalogs: true
I20250111 02:57:19.857051 19322 raft_consensus.cc:738] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:19.857412 18624 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_0791d732a14a40bbadc0dadb66c22f0c" } modify_external_catalogs: true
I20250111 02:57:19.857623 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c02166674a140f29e9f655d09532ced" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:19.857914 19480 consensus_queue.cc:1035] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:19.859550 19054 raft_consensus.cc:1270] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 2 FOLLOWER]: Refusing update from remote peer d0cedab82b154d62b78809edc6f28bfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250111 02:57:19.859910 19054 pending_rounds.cc:77] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f: Aborting all ops after (but not including) 0
I20250111 02:57:19.859951 19321 tablet_bootstrap.cc:492] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:19.860159 19054 pending_rounds.cc:99] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250111 02:57:19.860364 19321 ts_tablet_manager.cc:1397] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.114s user 0.003s sys 0.006s
I20250111 02:57:19.860419 19054 raft_consensus.cc:2883] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250111 02:57:19.833278 19326 log.cc:1198] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/6a2234755d474740a55184660119eb11
I20250111 02:57:19.862840 19321 raft_consensus.cc:357] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.863814 19326 ts_tablet_manager.cc:1939] T 6a2234755d474740a55184660119eb11 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:19.871289 19480 raft_consensus.cc:491] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:19.871807 19480 raft_consensus.cc:513] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.875094 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5118d4b9cfbb49e8af6af56180858336" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:19.877646 19480 leader_election.cc:290] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:19.881505 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet ede92343d9b54c29b8173a5eafcf68c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 199ca8692f39445f93d8c27e2134a32c) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:19.863972 19401 consensus_queue.cc:1035] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20250111 02:57:19.880343 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5118d4b9cfbb49e8af6af56180858336" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:19.865283 19325 raft_consensus.cc:2267] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:19.885277 19026 tablet_service.cc:1514] Processing DeleteTablet for tablet 32afc2f158254811836fcf00f7f409ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:19 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:19.886112 19325 ts_tablet_manager.cc:1905] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:19.890946 18986 leader_election.cc:343] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:19.893399 18625 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_293ee09817814cafa48fc9667e47f262" } new_table_name: "default.Table_53d44be8eaeb4e59a8e3454aac56daca" modify_external_catalogs: true
I20250111 02:57:19.863602 19321 raft_consensus.cc:383] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:19.894026 19321 raft_consensus.cc:738] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:19.894907 19321 consensus_queue.cc:260] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.857686 19322 consensus_queue.cc:260] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.901706 18908 raft_consensus.cc:2463] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:19.902894 19322 ts_tablet_manager.cc:1428] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
I20250111 02:57:19.904124 18989 leader_election.cc:304] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters: d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:19.905550 19480 raft_consensus.cc:2798] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:19.905972 19480 raft_consensus.cc:491] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:19.906324 19480 raft_consensus.cc:3054] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:19.910866 19321 ts_tablet_manager.cc:1428] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.050s user 0.006s sys 0.000s
I20250111 02:57:19.916923 19322 tablet_bootstrap.cc:492] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:19.924190 19321 tablet_bootstrap.cc:492] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:19.929019 19330 tablet_bootstrap.cc:654] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:19.866077 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 79) took 1592 ms. Trace:
I20250111 02:57:19.945597 19327 tablet_replica.cc:331] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:19.949105 19322 tablet_bootstrap.cc:654] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:19.955834 19321 tablet_bootstrap.cc:654] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:19.956274 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 616655aa75824fffa46f81c4f95d5159 (table default.Table_a58de727de864b9a8ecab09048b72fb7 [id=581224f40f0c4f03a9224f06d660c883]) successfully deleted
W20250111 02:57:19.957070 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 92aa64b2de5746d1b12cc93a10c9d56e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92aa64b2de5746d1b12cc93a10c9d56e already in progress: creating tablet
W20250111 02:57:19.957815 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 32afc2f158254811836fcf00f7f409ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32afc2f158254811836fcf00f7f409ed already in progress: creating tablet
I20250111 02:57:19.970785 18626 master_service.cc:946] ReplaceTablet: received request to replace tablet 32afc2f158254811836fcf00f7f409ed from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:19.971300 19444 raft_consensus.cc:513] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.972450 19195 raft_consensus.cc:1235] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [term 2 FOLLOWER]: Rejecting Update request from peer 54628fbcec2f4359aa9e26a734d4d29f for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250111 02:57:19.975824 19444 leader_election.cc:290] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 2 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:19.976807 19321 tablet_bootstrap.cc:492] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:19.977298 19321 ts_tablet_manager.cc:1397] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.005s
I20250111 02:57:19.983616 19322 tablet_bootstrap.cc:492] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:19.984047 19322 ts_tablet_manager.cc:1397] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.001s
I20250111 02:57:19.986347 19322 raft_consensus.cc:357] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.992617 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_e7b6efb1d2ad4fefbfa0732b863704eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:19.941639 19326 rpcz_store.cc:276] 0111 02:57:18.273668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.275129 (+ 1461us) service_pool.cc:225] Handling call
0111 02:57:18.299140 (+ 24011us) ts_tablet_manager.cc:1145] Deleting tablet 6a2234755d474740a55184660119eb11
0111 02:57:18.299226 (+ 86us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:18.423819 (+124593us) spinlock_profiling.cc:243] Waited 50.3 ms on lock 0x7b58000d04b0. stack: 00007f93b0e01853 00007f93b0e01750 00007f93b0976584 00000000004fcd24 00000000004fccb9 00007f93b7d016d1 00007f93b7cf81c3 00007f93b7d5baba 00007f93b7d84375 00007f93b81a11ac 00007f93b8db1d5a 00007f93b8de1ca0 00007f93b8dc8e6d 00007f93b8dc8e19 00007f93b8dc8da9 00007f93b8dc8d71
0111 02:57:19.487760 (+1063941us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:19.804601 (+316841us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:19.865998 (+ 61397us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51184,"spinlock_wait_cycles":115591936,"tablet-delete.queue_time_us":22457}
I20250111 02:57:19.997774 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 83) took 1722 ms. Trace:
I20250111 02:57:19.997514 19480 raft_consensus.cc:513] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:19.998142 19326 rpcz_store.cc:276] 0111 02:57:18.275281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.282132 (+ 6851us) service_pool.cc:225] Handling call
0111 02:57:19.997362 (+1715230us) ts_tablet_manager.cc:1145] Deleting tablet ef70b417568049f0b496f4d8e5f7e0e4
0111 02:57:19.997425 (+ 63us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:19.997713 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1714340}
I20250111 02:57:19.990828 19451 raft_consensus.cc:491] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:19.999590 19326 tablet_replica.cc:331] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:19.999475 19451 raft_consensus.cc:513] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.000676 19326 raft_consensus.cc:2238] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:20.001447 19451 leader_election.cc:290] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:20.002827 19326 pending_rounds.cc:62] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f: Trying to abort 1 pending ops.
I20250111 02:57:20.003166 19326 pending_rounds.cc:69] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112967117168398336 op_type: NO_OP noop_request { }
I20250111 02:57:20.003686 19326 raft_consensus.cc:2883] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:57:20.004204 19326 raft_consensus.cc:2267] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250111 02:57:20.005050 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7b6efb1d2ad4fefbfa0732b863704eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:19.982960 19427 raft_consensus.cc:513] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:20.009650 18989 leader_election.cc:343] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 616655aa75824fffa46f81c4f95d5159
I20250111 02:57:20.012681 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 6a2234755d474740a55184660119eb11 (table default.Table_f05e5d6100bc46379e8598ce5d367e03 [id=9493f23c2c874bda94417f245e37e24f]) successfully deleted
W20250111 02:57:20.013703 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet ef70b417568049f0b496f4d8e5f7e0e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef70b417568049f0b496f4d8e5f7e0e4 already in progress: creating tablet
I20250111 02:57:20.016717 19480 leader_election.cc:290] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:20.023151 18624 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 5f7d810675354a5c84b02ea41eef99a8
W20250111 02:57:20.026901 18625 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ffd063f8889b4bcdb9656466f0b144f3 has no consensus state
I20250111 02:57:20.009364 19327 raft_consensus.cc:2238] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:20.034070 19327 raft_consensus.cc:2267] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:20.019148 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5118d4b9cfbb49e8af6af56180858336" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:20.035178 18909 raft_consensus.cc:3054] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:20.038278 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede92343d9b54c29b8173a5eafcf68c6" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
W20250111 02:57:20.039893 18842 leader_election.cc:343] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:19.987028 19322 raft_consensus.cc:383] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:20.044260 19322 raft_consensus.cc:738] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:20.044329 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_39f0114c91ef4d1c86ef0cf522e31e11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:20.045207 19330 tablet_bootstrap.cc:492] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:20.045006 19322 consensus_queue.cc:260] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.045624 19330 ts_tablet_manager.cc:1397] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.220s user 0.009s sys 0.005s
I20250111 02:57:20.043701 19321 raft_consensus.cc:357] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.046550 19321 raft_consensus.cc:383] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:20.046834 19321 raft_consensus.cc:738] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:20.047503 19427 leader_election.cc:290] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:20.048574 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39f0114c91ef4d1c86ef0cf522e31e11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:20.053079 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_f36240205653470e9a67400b787ce860"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:20.058297 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f36240205653470e9a67400b787ce860 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:20.059484 19330 raft_consensus.cc:357] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.060253 19330 raft_consensus.cc:383] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:20.060523 19330 raft_consensus.cc:738] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:20.061205 19330 consensus_queue.cc:260] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.065322 19330 ts_tablet_manager.cc:1428] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20250111 02:57:20.066232 19330 tablet_bootstrap.cc:492] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:20.071099 19326 ts_tablet_manager.cc:1905] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:20.076494 19322 ts_tablet_manager.cc:1428] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.092s user 0.001s sys 0.005s
I20250111 02:57:20.079944 18909 raft_consensus.cc:2463] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:20.094110 19330 tablet_bootstrap.cc:654] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:20.096613 19327 ts_tablet_manager.cc:1905] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:20.107365 18989 leader_election.cc:304] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:20.108356 19451 raft_consensus.cc:2798] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:20.108851 19451 raft_consensus.cc:695] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:20.109690 19451 consensus_queue.cc:237] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.113274 19322 tablet_bootstrap.cc:492] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:20.132124 19330 tablet_bootstrap.cc:492] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:20.132628 19330 ts_tablet_manager.cc:1397] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.001s
I20250111 02:57:20.135015 19330 raft_consensus.cc:357] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.142817 19330 raft_consensus.cc:383] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:20.143112 19330 raft_consensus.cc:738] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:20.143676 19330 consensus_queue.cc:260] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.157156 19321 consensus_queue.cc:260] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.167981 19325 ts_tablet_manager.cc:1918] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:20.168658 19325 log.cc:1198] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/72bcbe140a82462d909bd6d41bfbd2af
I20250111 02:57:20.169497 19325 ts_tablet_manager.cc:1939] T 72bcbe140a82462d909bd6d41bfbd2af P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:20.173306 19322 tablet_bootstrap.cc:654] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:20.191435 19330 ts_tablet_manager.cc:1428] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.058s user 0.006s sys 0.002s
I20250111 02:57:20.193059 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c02166674a140f29e9f655d09532ced" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:20.197630 19326 ts_tablet_manager.cc:1918] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:20.200038 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_4f96c1244592444a994b5c5631eaaadc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:20.201829 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f96c1244592444a994b5c5631eaaadc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:20.203876 19444 consensus_queue.cc:1035] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:20.205932 19480 consensus_queue.cc:1035] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:20.198478 19330 tablet_bootstrap.cc:492] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:20.221648 19321 ts_tablet_manager.cc:1428] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.244s user 0.009s sys 0.003s
I20250111 02:57:20.200485 19326 log.cc:1198] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/4feb96efdfc44fe4a7cd46dc8deb4fe7
I20250111 02:57:20.226822 18625 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_154c30276c8d4a8e805dc2fb85867bbf" } modify_external_catalogs: true
I20250111 02:57:20.227561 18625 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_154c30276c8d4a8e805dc2fb85867bbf" } modify_external_catalogs: true
I20250111 02:57:20.228067 19326 ts_tablet_manager.cc:1939] T 4feb96efdfc44fe4a7cd46dc8deb4fe7 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:20.229893 19480 raft_consensus.cc:491] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:20.230370 19480 raft_consensus.cc:513] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.233239 18628 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_053b33f398ad460e8e4cae9a9a80b13d" } new_table_name: "default.Table_ef8a731a05e841b6bd31bd3df1a8e375" modify_external_catalogs: true
I20250111 02:57:20.234576 19480 leader_election.cc:290] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:20.239739 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede92343d9b54c29b8173a5eafcf68c6" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:20.241235 18989 leader_election.cc:343] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: Tablet not RUNNING: STOPPED
I20250111 02:57:20.246066 19330 tablet_bootstrap.cc:654] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:20.248802 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 80) took 1974 ms. Trace:
I20250111 02:57:20.252902 19326 rpcz_store.cc:276] 0111 02:57:18.274074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.287881 (+ 13807us) service_pool.cc:225] Handling call
0111 02:57:19.999048 (+1711167us) ts_tablet_manager.cc:1145] Deleting tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7
0111 02:57:19.999467 (+ 419us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:20.111594 (+112127us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:20.197407 (+ 85813us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:20.248740 (+ 51333us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":291712,"tablet-delete.queue_time_us":1710352}
I20250111 02:57:20.254606 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 81) took 1980 ms. Trace:
I20250111 02:57:20.254680 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 (table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0]) successfully deleted
I20250111 02:57:20.255000 19326 rpcz_store.cc:276] 0111 02:57:18.274449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.287213 (+ 12764us) service_pool.cc:225] Handling call
0111 02:57:20.254318 (+1967105us) ts_tablet_manager.cc:1145] Deleting tablet 0d6aa50a4dfe4398a82877b6dfcb5c29
0111 02:57:20.254364 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:20.254554 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1963411}
W20250111 02:57:20.263947 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 because the tablet was not found. No further retry: Not found: Tablet not found: 0d6aa50a4dfe4398a82877b6dfcb5c29
I20250111 02:57:20.264523 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 78) took 1991 ms. Trace:
I20250111 02:57:20.264896 19326 rpcz_store.cc:276] 0111 02:57:18.273176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.293185 (+ 20009us) service_pool.cc:225] Handling call
0111 02:57:20.264030 (+1970845us) ts_tablet_manager.cc:1145] Deleting tablet c28958a1018344a48d18e6c0f03cd0a3
0111 02:57:20.264075 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:20.264462 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1969935}
I20250111 02:57:20.266278 19326 tablet_replica.cc:331] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:20.227133 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 82) took 2121 ms. Trace:
W20250111 02:57:20.267076 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet c28958a1018344a48d18e6c0f03cd0a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c28958a1018344a48d18e6c0f03cd0a3 already in progress: creating tablet
I20250111 02:57:20.267047 19325 rpcz_store.cc:276] 0111 02:57:18.105455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.105586 (+ 131us) service_pool.cc:225] Handling call
0111 02:57:19.845749 (+1740163us) ts_tablet_manager.cc:1145] Deleting tablet 72bcbe140a82462d909bd6d41bfbd2af
0111 02:57:19.845792 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:20.009724 (+163932us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:20.167538 (+157814us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:20.227003 (+ 59465us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1739606}
I20250111 02:57:20.268093 19321 tablet_bootstrap.cc:492] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:20.268524 19325 tablet_replica.cc:331] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:20.269263 19325 raft_consensus.cc:2238] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:20.269848 19325 raft_consensus.cc:2267] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:20.272441 19325 ts_tablet_manager.cc:1905] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:20.275321 19326 raft_consensus.cc:2238] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:20.275945 19326 raft_consensus.cc:2267] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:20.278429 19327 ts_tablet_manager.cc:1918] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:20.278815 19327 log.cc:1198] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/ede92343d9b54c29b8173a5eafcf68c6
I20250111 02:57:20.279610 19327 ts_tablet_manager.cc:1939] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:20.279520 18890 tablet_service.cc:1467] Processing CreateTablet for tablet 62c6b166a29943ceb15f093fac728c79 (DEFAULT_TABLE table=default.Table_8118351b275b46018f91df033a1270db [id=219736e784e4473a96064a135a667525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.280678 18890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62c6b166a29943ceb15f093fac728c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.285302 18622 catalog_manager.cc:5526] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa reported cstate change: term changed from 1 to 2, leader changed from 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66) to d0cedab82b154d62b78809edc6f28bfa (127.18.20.67). New cstate: current_term: 2 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } }
I20250111 02:57:20.288372 18622 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 5baef8688ede42fca28cfb6b0bdeefa6 (table default.Table_7f4578da9f5e418c9d67a535471b45cd [id=154c37dc0dba44bdab5e0315756dc7cf]). Expected version 1 got 0
I20250111 02:57:20.289095 18622 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet b5148fa2e6504561bd1b1a19e48993d7 (table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0]). Expected version 1 got 0
I20250111 02:57:20.290463 18626 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_b6421db891c148c688d25610559fe16f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:20.292315 18626 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6421db891c148c688d25610559fe16f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:20.304899 18886 tablet_service.cc:1467] Processing CreateTablet for tablet ffd063f8889b4bcdb9656466f0b144f3 (DEFAULT_TABLE table=default.Table_53d44be8eaeb4e59a8e3454aac56daca [id=8de61815471842e593a7d29efffc443c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.306027 18886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffd063f8889b4bcdb9656466f0b144f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.292384 19025 tablet_service.cc:1467] Processing CreateTablet for tablet 62c6b166a29943ceb15f093fac728c79 (DEFAULT_TABLE table=default.Table_8118351b275b46018f91df033a1270db [id=219736e784e4473a96064a135a667525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.307613 19025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62c6b166a29943ceb15f093fac728c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.316588 19330 tablet_bootstrap.cc:492] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:20.317150 19330 ts_tablet_manager.cc:1397] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.125s user 0.010s sys 0.001s
I20250111 02:57:20.319792 19330 raft_consensus.cc:357] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.324955 19330 raft_consensus.cc:383] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250111 02:57:20.329080 18628 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4bd42ad66064bcaa043746081f1c759 consensus state has no leader
I20250111 02:57:20.332104 19322 tablet_bootstrap.cc:492] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:20.332576 19322 ts_tablet_manager.cc:1397] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.220s user 0.011s sys 0.003s
I20250111 02:57:20.332891 18625 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 0dbb63ab78f84995ae6d9fb2aae78dcf
I20250111 02:57:20.333914 19330 raft_consensus.cc:738] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:20.324630 19326 ts_tablet_manager.cc:1905] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:20.335943 19029 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dbb63ab78f84995ae6d9fb2aae78dcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:20.337320 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet ede92343d9b54c29b8173a5eafcf68c6 (table default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]) successfully deleted
I20250111 02:57:20.337720 18880 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dbb63ab78f84995ae6d9fb2aae78dcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:20.339437 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_dc22147bd8ba4ea18bc4f41b25b277c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:20.341925 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc22147bd8ba4ea18bc4f41b25b277c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:20.339454 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 0dbb63ab78f84995ae6d9fb2aae78dcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dbb63ab78f84995ae6d9fb2aae78dcf already in progress: creating tablet
I20250111 02:57:20.347539 19330 consensus_queue.cc:260] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.334968 19322 raft_consensus.cc:357] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:20.348917 19322 raft_consensus.cc:383] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:20.349207 19322 raft_consensus.cc:738] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:20.352603 19322 consensus_queue.cc:260] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:20.364336 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c02166674a140f29e9f655d09532ced" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:20.366806 18908 raft_consensus.cc:1270] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Refusing update from remote peer d0cedab82b154d62b78809edc6f28bfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250111 02:57:20.370002 18880 tablet_service.cc:1467] Processing CreateTablet for tablet 220da826a0f140d59193e8ad72096b60 (DEFAULT_TABLE table=default.Table_346b3997f77e4cfdb856142115ee0efc [id=9143d1d3b8754ac78d615a1cc4601d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.371114 19321 tablet_bootstrap.cc:654] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:20.379171 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 220da826a0f140d59193e8ad72096b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.380656 19322 ts_tablet_manager.cc:1428] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.048s user 0.006s sys 0.001s
I20250111 02:57:20.383244 18630 catalog_manager.cc:5526] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } }
I20250111 02:57:20.384675 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet b5148fa2e6504561bd1b1a19e48993d7 (table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0]). Expected version 1 got 0
I20250111 02:57:20.385385 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet b8e8c72c99ac4c369957b571d74baffe (table default.Table_35d7f864912b40bcb5251a90d40c7849 [id=772d994bee2d4e34bd48912acecf8d48]). Expected version 1 got 0
I20250111 02:57:20.385955 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet 5baef8688ede42fca28cfb6b0bdeefa6 (table default.Table_7f4578da9f5e418c9d67a535471b45cd [id=154c37dc0dba44bdab5e0315756dc7cf]). Expected version 1 got 0
I20250111 02:57:20.386338 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_c0ddb0fbf2f347c0b82acf9060722ed2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:20.367594 19029 tablet_service.cc:1467] Processing CreateTablet for tablet ffd063f8889b4bcdb9656466f0b144f3 (DEFAULT_TABLE table=default.Table_53d44be8eaeb4e59a8e3454aac56daca [id=8de61815471842e593a7d29efffc443c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.387871 19029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffd063f8889b4bcdb9656466f0b144f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.388559 19330 ts_tablet_manager.cc:1428] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.071s user 0.005s sys 0.000s
I20250111 02:57:20.389464 19330 tablet_bootstrap.cc:492] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
W20250111 02:57:20.390419 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0ddb0fbf2f347c0b82acf9060722ed2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:20.392141 19529 raft_consensus.cc:491] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:20.392628 19529 raft_consensus.cc:513] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.394613 19529 leader_election.cc:290] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:20.405280 19322 tablet_bootstrap.cc:492] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:20.405908 18621 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 3c02166674a140f29e9f655d09532ced
I20250111 02:57:20.407622 18621 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47522 (request call id 146) took 1206 ms. Trace:
I20250111 02:57:20.412087 19427 raft_consensus.cc:491] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:20.412581 19427 raft_consensus.cc:513] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.414566 19427 leader_election.cc:290] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:20.416374 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066c87476a854e169fd4f5f1cdbb49f7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:20.417006 19054 raft_consensus.cc:2463] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:20.418198 18842 leader_election.cc:304] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:20.408036 18621 rpcz_store.cc:276] 0111 02:57:19.200812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:19.289118 (+ 88306us) service_pool.cc:225] Handling call
0111 02:57:19.535877 (+246759us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d53616 00007fd427abbb44 00007fd427abbad2 00007fd427abba01 00007fd427abb97c 00007fd427abac32 00007fd425d79dfc 00007fd425d79236
0111 02:57:19.537086 (+ 1209us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0111 02:57:19.537621 (+ 535us) catalog_manager.cc:2753] Modifying in-memory table state
0111 02:57:19.537765 (+ 144us) catalog_manager.cc:2762] Locking tablets
0111 02:57:19.546177 (+ 8412us) catalog_manager.cc:2776] Removing table and tablets from system table
0111 02:57:19.915473 (+369296us) catalog_manager.cc:2794] Removing table from by-name map
0111 02:57:19.915745 (+ 272us) catalog_manager.cc:2810] Committing in-memory state
0111 02:57:20.405790 (+490045us) catalog_manager.cc:2814] Aborting table tasks
0111 02:57:20.407549 (+ 1759us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:19.728965 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:19.729312 (+ 347us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:19.729338 (+ 26us) write_op.cc:435] Acquired schema lock
0111 02:57:19.729351 (+ 13us) tablet.cc:659] Decoding operations
0111 02:57:19.729655 (+ 304us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:19.729695 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:19.729717 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:19.729979 (+ 262us) tablet.cc:698] Row locks acquired
0111 02:57:19.729991 (+ 12us) write_op.cc:260] PREPARE: finished
0111 02:57:19.730092 (+ 101us) write_op.cc:270] Start()
0111 02:57:19.730186 (+ 94us) write_op.cc:276] Timestamp: P: 1736564239730046 usec, L: 0
0111 02:57:19.730200 (+ 14us) op_driver.cc:348] REPLICATION: starting
0111 02:57:19.767553 (+ 37353us) log.cc:844] Serialized 728 byte log entry
0111 02:57:19.776958 (+ 9405us) spinlock_profiling.cc:243] Waited 4.56 ms on lock 0x7b2000001078. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f316d1 00007fd427f8c1f5 00007fd427fa81b1 00007fd4284143b1 00007fd428413b88 00007fd428418821 00007fd4284187d9 00007fd428418769 00007fd428418731 00007fd42841795d 000000000059ea04
0111 02:57:19.913562 (+136604us) op_driver.cc:464] REPLICATION: finished
0111 02:57:19.913805 (+ 243us) write_op.cc:301] APPLY: starting
0111 02:57:19.913863 (+ 58us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:19.913974 (+ 111us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:19.913994 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:19.914232 (+ 238us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:19.914384 (+ 152us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0111 02:57:19.914433 (+ 49us) write_op.cc:312] APPLY: finished
0111 02:57:19.914634 (+ 201us) log.cc:844] Serialized 29 byte log entry
0111 02:57:19.914775 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:19.914845 (+ 70us) write_op.cc:454] Released schema lock
0111 02:57:19.914876 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:19.915144 (+ 268us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:19.915169 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":565746944,"child_traces":[["op",{"apply.queue_time_us":98,"apply.run_cpu_time_us":1280,"apply.run_wall_time_us":1279,"mutex_wait_us":34,"num_ops":2,"prepare.queue_time_us":170536,"prepare.run_cpu_time_us":3007,"prepare.run_wall_time_us":48085,"raft.queue_time_us":125,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":15,"replication_time_us":183261,"spinlock_wait_cycles":10492160}]]}
I20250111 02:57:20.419597 19427 raft_consensus.cc:2798] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:20.419929 19427 raft_consensus.cc:491] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:20.420233 19427 raft_consensus.cc:3054] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:20.423732 19025 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c02166674a140f29e9f655d09532ced with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:19 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:20.424358 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c02166674a140f29e9f655d09532ced with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:19 UTC) from {username='slave'} at 127.0.0.1:51082
W20250111 02:57:20.416347 18622 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.67:53171 (request call id 25) took 1078 ms (client timeout 1000 ms). Trace:
I20250111 02:57:20.427006 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede92343d9b54c29b8173a5eafcf68c6" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:20.427863 19195 raft_consensus.cc:3054] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:20.425149 18622 rpcz_store.cc:269] 0111 02:57:19.338284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:19.357496 (+ 19212us) service_pool.cc:225] Handling call
0111 02:57:19.539668 (+182172us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d51dee 00007fd427aaffc4 00007fd427aaff52 00007fd427aafe81 00007fd427aafdfc 00007fd427aaf0b2 00007fd425d79dfc 00007fd425d79236
0111 02:57:20.416263 (+876595us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:20.293375 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:20.293772 (+ 397us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:20.293798 (+ 26us) write_op.cc:435] Acquired schema lock
0111 02:57:20.293813 (+ 15us) tablet.cc:659] Decoding operations
0111 02:57:20.294083 (+ 270us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:20.294124 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:20.294150 (+ 26us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:20.294311 (+ 161us) tablet.cc:698] Row locks acquired
0111 02:57:20.294327 (+ 16us) write_op.cc:260] PREPARE: finished
0111 02:57:20.294443 (+ 116us) write_op.cc:270] Start()
0111 02:57:20.294532 (+ 89us) write_op.cc:276] Timestamp: P: 1736564240294408 usec, L: 0
0111 02:57:20.294552 (+ 20us) op_driver.cc:348] REPLICATION: starting
0111 02:57:20.295097 (+ 545us) log.cc:844] Serialized 530 byte log entry
0111 02:57:20.373285 (+ 78188us) op_driver.cc:464] REPLICATION: finished
0111 02:57:20.373812 (+ 527us) write_op.cc:301] APPLY: starting
0111 02:57:20.373871 (+ 59us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:20.373964 (+ 93us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:20.374062 (+ 98us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:20.374251 (+ 189us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:20.374446 (+ 195us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:20.374509 (+ 63us) write_op.cc:312] APPLY: finished
0111 02:57:20.374683 (+ 174us) log.cc:844] Serialized 23 byte log entry
0111 02:57:20.374825 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:20.374886 (+ 61us) write_op.cc:454] Released schema lock
0111 02:57:20.374915 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:20.375156 (+ 241us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:20.375179 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":14486,"reported_tablets":5,"spinlock_wait_cycles":409497856,"child_traces":[["op",{"apply.queue_time_us":378,"apply.run_cpu_time_us":1317,"apply.run_wall_time_us":1317,"num_ops":1,"prepare.queue_time_us":2238,"prepare.run_cpu_time_us":2032,"prepare.run_wall_time_us":2032,"raft.queue_time_us":177,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":12,"replication_time_us":78627}]]}
I20250111 02:57:20.432536 19327 tablet_replica.cc:331] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:20.433410 19327 raft_consensus.cc:2238] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:20.433950 19327 raft_consensus.cc:2267] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
W20250111 02:57:20.435259 18627 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a983a670c2104398ac1a159ac3a1da7c consensus state has no leader
I20250111 02:57:20.436091 19195 raft_consensus.cc:2463] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:20.436630 19327 ts_tablet_manager.cc:1905] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:20.438288 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066c87476a854e169fd4f5f1cdbb49f7" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:20.438959 19052 raft_consensus.cc:2463] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:20.440182 19131 leader_election.cc:304] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:20.441164 19529 raft_consensus.cc:2798] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:20.441550 19529 raft_consensus.cc:491] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:20.441891 19529 raft_consensus.cc:3054] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:20.447665 19133 leader_election.cc:304] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:20.449298 19533 consensus_queue.cc:1046] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:20.427274 19427 raft_consensus.cc:513] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.450150 19533 raft_consensus.cc:3049] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:57:20.424958 19330 tablet_bootstrap.cc:654] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:20.450438 19533 raft_consensus.cc:738] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:20.451232 19533 consensus_queue.cc:260] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.452520 19533 raft_consensus.cc:3054] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Advancing to term 2
W20250111 02:57:20.473500 18615 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet a983a670c2104398ac1a159ac3a1da7c (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet a983a670c2104398ac1a159ac3a1da7c consensus state has no leader
I20250111 02:57:20.480533 19444 raft_consensus.cc:2743] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250111 02:57:20.481209 19133 consensus_peers.cc:487] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b715e32011114e6286c0b8e5406258c4. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:20.483137 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 5baef8688ede42fca28cfb6b0bdeefa6 (table default.Table_7f4578da9f5e418c9d67a535471b45cd [id=154c37dc0dba44bdab5e0315756dc7cf]). Expected version 1 got 0
I20250111 02:57:20.485499 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 72bcbe140a82462d909bd6d41bfbd2af (table default.Table_8d238bdc5f53451aab66467655c6fb2b [id=f35e4781910c45ba93c35a24da2b83c1]) successfully deleted
I20250111 02:57:20.491315 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet b5148fa2e6504561bd1b1a19e48993d7 (table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0]). Expected version 1 got 0
I20250111 02:57:20.497996 19326 ts_tablet_manager.cc:1918] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:20.498494 19326 log.cc:1198] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/8bfad6c08b4b44c1a664cda3b43acbb1
I20250111 02:57:20.499373 19326 ts_tablet_manager.cc:1939] T 8bfad6c08b4b44c1a664cda3b43acbb1 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:20.501415 19533 consensus_queue.cc:1035] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:20.501737 19029 tablet_service.cc:1467] Processing CreateTablet for tablet 220da826a0f140d59193e8ad72096b60 (DEFAULT_TABLE table=default.Table_346b3997f77e4cfdb856142115ee0efc [id=9143d1d3b8754ac78d615a1cc4601d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.503077 19029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 220da826a0f140d59193e8ad72096b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.502564 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_c398ec562f594aca8c1e62cde38c04f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:20.504191 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c398ec562f594aca8c1e62cde38c04f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:20.505980 19025 tablet_service.cc:1467] Processing CreateTablet for tablet ea76b354a75041e888cf862250e4947b (DEFAULT_TABLE table=default.Table_eaa40a643c0e43b9a1204616e7403efa [id=ec5cf5f9b6234008914c0ed664260dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.507539 19025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea76b354a75041e888cf862250e4947b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.513468 19330 tablet_bootstrap.cc:492] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:20.513899 19330 ts_tablet_manager.cc:1397] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.125s user 0.007s sys 0.007s
I20250111 02:57:20.515326 19024 tablet_service.cc:1514] Processing DeleteTablet for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 with delete_type TABLET_DATA_DELETED (replaced by tablet f1802e7349934fbb9d5f7eda37c3df20) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:20.522135 19024 tablet_service.cc:1514] Processing DeleteTablet for tablet 066c87476a854e169fd4f5f1cdbb49f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:20.522559 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet 5118d4b9cfbb49e8af6af56180858336 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:20.501432 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 84) took 2037 ms. Trace:
I20250111 02:57:20.525614 19326 rpcz_store.cc:276] 0111 02:57:18.463644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.464407 (+ 763us) service_pool.cc:225] Handling call
0111 02:57:20.266128 (+1801721us) ts_tablet_manager.cc:1145] Deleting tablet 8bfad6c08b4b44c1a664cda3b43acbb1
0111 02:57:20.266169 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:20.418075 (+151906us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:20.497754 (+ 79679us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:20.501372 (+ 3618us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1801049}
I20250111 02:57:20.526842 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 8bfad6c08b4b44c1a664cda3b43acbb1 (table default.Table_65af1591521e4974a28304ce51fe6fb1 [id=7cf2761276494576a4c2be6239374334]) successfully deleted
I20250111 02:57:20.449401 19529 raft_consensus.cc:513] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.486850 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5118d4b9cfbb49e8af6af56180858336" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:20.528978 19196 raft_consensus.cc:3054] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:20.532919 19529 leader_election.cc:290] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:20.532832 18625 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_346b3997f77e4cfdb856142115ee0efc" } new_table_name: "default.Table_53adc14208674563b5bda158cf0c68a3" modify_external_catalogs: true
I20250111 02:57:20.533706 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066c87476a854e169fd4f5f1cdbb49f7" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:20.534257 19052 raft_consensus.cc:3054] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:20.537407 19327 ts_tablet_manager.cc:1918] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:20.537497 18627 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_61c40191c611402da3829f2ef5379f61" } modify_external_catalogs: true
I20250111 02:57:20.537762 19327 log.cc:1198] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/3c02166674a140f29e9f655d09532ced
I20250111 02:57:20.538043 18627 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_61c40191c611402da3829f2ef5379f61" } modify_external_catalogs: true
I20250111 02:57:20.538245 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 89) took 1565 ms. Trace:
I20250111 02:57:20.538477 19327 ts_tablet_manager.cc:1939] T 3c02166674a140f29e9f655d09532ced P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:20.542431 19033 tablet_service.cc:1467] Processing CreateTablet for tablet e46894710a72407993b36801f3b90eb0 (DEFAULT_TABLE table=default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.538549 19326 rpcz_store.cc:276] 0111 02:57:18.972546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.972726 (+ 180us) service_pool.cc:225] Handling call
0111 02:57:20.528108 (+1555382us) ts_tablet_manager.cc:1145] Deleting tablet 9fe4e60db4244a9a9696153b630c575b
0111 02:57:20.528151 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:20.528409 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1554778}
I20250111 02:57:20.543632 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e46894710a72407993b36801f3b90eb0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:20.544297 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 9fe4e60db4244a9a9696153b630c575b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fe4e60db4244a9a9696153b630c575b already in progress: creating tablet
I20250111 02:57:20.544785 18890 tablet_service.cc:1467] Processing CreateTablet for tablet ea76b354a75041e888cf862250e4947b (DEFAULT_TABLE table=default.Table_eaa40a643c0e43b9a1204616e7403efa [id=ec5cf5f9b6234008914c0ed664260dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.546342 18884 tablet_service.cc:1467] Processing CreateTablet for tablet e46894710a72407993b36801f3b90eb0 (DEFAULT_TABLE table=default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.547124 19024 tablet_service.cc:1467] Processing CreateTablet for tablet 237ea33846044127987c1d81a3203553 (DEFAULT_TABLE table=default.Table_b9769d513090402bb596c949defad0df [id=76e72ea89f6e408fbaeda0efc7e79770]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.547715 18879 tablet_service.cc:1467] Processing CreateTablet for tablet 237ea33846044127987c1d81a3203553 (DEFAULT_TABLE table=default.Table_b9769d513090402bb596c949defad0df [id=76e72ea89f6e408fbaeda0efc7e79770]), partition=RANGE (key) PARTITION UNBOUNDED
W20250111 02:57:20.547989 18614 catalog_manager.cc:4945] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): alter failed for tablet 0d6aa50a4dfe4398a82877b6dfcb5c29 (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]),no further retry: Not found: Tablet not found: 0d6aa50a4dfe4398a82877b6dfcb5c29
I20250111 02:57:20.548243 19024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 237ea33846044127987c1d81a3203553. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.488654 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616655aa75824fffa46f81c4f95d5159" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:20.489491 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede92343d9b54c29b8173a5eafcf68c6" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:20.556751 19197 raft_consensus.cc:2388] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54628fbcec2f4359aa9e26a734d4d29f in current term 1: Already voted for candidate 8221604733fe4fc0a1fc32de8d6af71c in this term.
I20250111 02:57:20.492404 19174 tablet_service.cc:1467] Processing CreateTablet for tablet 62c6b166a29943ceb15f093fac728c79 (DEFAULT_TABLE table=default.Table_8118351b275b46018f91df033a1270db [id=219736e784e4473a96064a135a667525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.558218 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62c6b166a29943ceb15f093fac728c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.562464 19171 tablet_service.cc:1467] Processing CreateTablet for tablet ea76b354a75041e888cf862250e4947b (DEFAULT_TABLE table=default.Table_eaa40a643c0e43b9a1204616e7403efa [id=ec5cf5f9b6234008914c0ed664260dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.494343 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dbb63ab78f84995ae6d9fb2aae78dcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:20.496517 19176 tablet_service.cc:1467] Processing CreateTablet for tablet ffd063f8889b4bcdb9656466f0b144f3 (DEFAULT_TABLE table=default.Table_53d44be8eaeb4e59a8e3454aac56daca [id=8de61815471842e593a7d29efffc443c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.498194 19169 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c02166674a140f29e9f655d09532ced with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:19 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:20.499261 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet f610a4ff1cc34a928f2db73f14834b25 with delete_type TABLET_DATA_DELETED (replaced by tablet 855dac47872d4c5fbf8f5e38b588c7fe) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:20.500535 19167 tablet_service.cc:1467] Processing CreateTablet for tablet 220da826a0f140d59193e8ad72096b60 (DEFAULT_TABLE table=default.Table_346b3997f77e4cfdb856142115ee0efc [id=9143d1d3b8754ac78d615a1cc4601d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.512300 19325 ts_tablet_manager.cc:1918] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:20.564234 19171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea76b354a75041e888cf862250e4947b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.564314 19325 log.cc:1198] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/6a2234755d474740a55184660119eb11
I20250111 02:57:20.565088 19325 ts_tablet_manager.cc:1939] T 6a2234755d474740a55184660119eb11 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:20.566382 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066c87476a854e169fd4f5f1cdbb49f7" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:20.567739 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 220da826a0f140d59193e8ad72096b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.571992 19322 tablet_bootstrap.cc:654] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:20.572211 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffd063f8889b4bcdb9656466f0b144f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.576853 18908 raft_consensus.cc:2388] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0cedab82b154d62b78809edc6f28bfa in current term 1: Already voted for candidate 8221604733fe4fc0a1fc32de8d6af71c in this term.
I20250111 02:57:20.578876 18881 tablet_service.cc:1467] Processing CreateTablet for tablet e4bd42ad66064bcaa043746081f1c759 (DEFAULT_TABLE table=default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.581847 19196 raft_consensus.cc:2463] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:20.582582 18890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea76b354a75041e888cf862250e4947b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.586813 18884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e46894710a72407993b36801f3b90eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.584513 19321 tablet_bootstrap.cc:492] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
W20250111 02:57:20.526975 19133 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 45993.8us
I20250111 02:57:20.588186 19321 ts_tablet_manager.cc:1397] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.320s user 0.013s sys 0.010s
I20250111 02:57:20.588840 19444 consensus_queue.cc:1035] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:20.588562 18622 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_75e5f05a97b14bf883909660868761fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:20.590972 18622 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75e5f05a97b14bf883909660868761fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:20.595098 18879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 237ea33846044127987c1d81a3203553. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.595695 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066c87476a854e169fd4f5f1cdbb49f7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:20.596518 19196 raft_consensus.cc:2388] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate d0cedab82b154d62b78809edc6f28bfa in this term.
I20250111 02:57:20.599869 18881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4bd42ad66064bcaa043746081f1c759. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:20.600970 18625 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 220da826a0f140d59193e8ad72096b60 consensus state has no leader
I20250111 02:57:20.602066 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 79) took 2529 ms. Trace:
I20250111 02:57:20.602416 19325 rpcz_store.cc:276] 0111 02:57:18.072582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.106645 (+ 34063us) service_pool.cc:225] Handling call
0111 02:57:20.268321 (+2161676us) ts_tablet_manager.cc:1145] Deleting tablet 6a2234755d474740a55184660119eb11
0111 02:57:20.268405 (+ 84us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:20.425537 (+157132us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:20.512046 (+ 86509us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:20.601995 (+ 89949us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2161089}
I20250111 02:57:20.603871 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 84) took 2382 ms. Trace:
I20250111 02:57:20.604033 18841 leader_election.cc:304] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f
I20250111 02:57:20.604202 19325 rpcz_store.cc:276] 0111 02:57:18.221856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.222004 (+ 148us) service_pool.cc:225] Handling call
0111 02:57:20.603474 (+2381470us) ts_tablet_manager.cc:1145] Deleting tablet ef70b417568049f0b496f4d8e5f7e0e4
0111 02:57:20.603523 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:20.603818 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2380856}
W20250111 02:57:20.604705 18841 raft_consensus.cc:2681] T ede92343d9b54c29b8173a5eafcf68c6 P 8221604733fe4fc0a1fc32de8d6af71c: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250111 02:57:20.605250 19325 tablet_replica.cc:331] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:20.605926 19325 raft_consensus.cc:2238] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:20.606786 19325 raft_consensus.cc:2267] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:20.607889 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066c87476a854e169fd4f5f1cdbb49f7" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:20.608495 18630 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.66:37049 (request call id 27) took 797 ms (client timeout 1000 ms). Trace:
I20250111 02:57:20.608669 18908 raft_consensus.cc:2388] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d0cedab82b154d62b78809edc6f28bfa in current term 1: Already voted for candidate 8221604733fe4fc0a1fc32de8d6af71c in this term.
I20250111 02:57:20.610777 19427 leader_election.cc:290] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:20.611213 18627 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 5cf1ce65ccf54358b95665b9764fbdb0
I20250111 02:57:20.613879 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet ede92343d9b54c29b8173a5eafcf68c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 199ca8692f39445f93d8c27e2134a32c) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:20.614665 19169 tablet_service.cc:1514] Processing DeleteTablet for tablet 066c87476a854e169fd4f5f1cdbb49f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:20.615370 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet 5118d4b9cfbb49e8af6af56180858336 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:20.617095 19168 tablet_service.cc:1467] Processing CreateTablet for tablet e46894710a72407993b36801f3b90eb0 (DEFAULT_TABLE table=default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.618801 19168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e46894710a72407993b36801f3b90eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.623626 18627 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_d37d3e82138442838c482c283e2d1002" } modify_external_catalogs: true
I20250111 02:57:20.624334 18627 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_d37d3e82138442838c482c283e2d1002" } modify_external_catalogs: true
I20250111 02:57:20.626906 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf1ce65ccf54358b95665b9764fbdb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:20.632056 19173 tablet_service.cc:1467] Processing CreateTablet for tablet 237ea33846044127987c1d81a3203553 (DEFAULT_TABLE table=default.Table_b9769d513090402bb596c949defad0df [id=76e72ea89f6e408fbaeda0efc7e79770]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.633302 19173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 237ea33846044127987c1d81a3203553. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.633688 18625 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet b5148fa2e6504561bd1b1a19e48993d7 (table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0]). Expected version 1 got 0
I20250111 02:57:20.644022 19325 ts_tablet_manager.cc:1905] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:20.634312 18625 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet f1802e7349934fbb9d5f7eda37c3df20 (table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0]). Expected version 4 got 0
W20250111 02:57:20.608835 18630 rpcz_store.cc:269] 0111 02:57:19.810691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:19.810856 (+ 165us) service_pool.cc:225] Handling call
0111 02:57:20.519050 (+708194us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:20.437993 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:20.438396 (+ 403us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:20.438419 (+ 23us) write_op.cc:435] Acquired schema lock
0111 02:57:20.438431 (+ 12us) tablet.cc:659] Decoding operations
0111 02:57:20.438690 (+ 259us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:20.438725 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:20.438794 (+ 69us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:20.438938 (+ 144us) tablet.cc:698] Row locks acquired
0111 02:57:20.438950 (+ 12us) write_op.cc:260] PREPARE: finished
0111 02:57:20.439103 (+ 153us) write_op.cc:270] Start()
0111 02:57:20.439192 (+ 89us) write_op.cc:276] Timestamp: P: 1736564240439039 usec, L: 0
0111 02:57:20.439205 (+ 13us) op_driver.cc:348] REPLICATION: starting
0111 02:57:20.439741 (+ 536us) log.cc:844] Serialized 530 byte log entry
0111 02:57:20.468238 (+ 28497us) op_driver.cc:464] REPLICATION: finished
0111 02:57:20.468764 (+ 526us) write_op.cc:301] APPLY: starting
0111 02:57:20.468808 (+ 44us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:20.468890 (+ 82us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:20.468968 (+ 78us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:20.469100 (+ 132us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:20.469245 (+ 145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:20.469287 (+ 42us) write_op.cc:312] APPLY: finished
0111 02:57:20.469470 (+ 183us) log.cc:844] Serialized 23 byte log entry
0111 02:57:20.469615 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:20.469676 (+ 61us) write_op.cc:454] Released schema lock
0111 02:57:20.469710 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:20.469924 (+ 214us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:20.469948 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":390,"apply.run_cpu_time_us":1125,"apply.run_wall_time_us":1124,"num_ops":1,"prepare.queue_time_us":116,"prepare.run_cpu_time_us":1863,"prepare.run_wall_time_us":1947,"replication_time_us":28957}]]}
I20250111 02:57:20.667395 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066c87476a854e169fd4f5f1cdbb49f7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:20.668023 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 90) took 1694 ms. Trace:
I20250111 02:57:20.668324 19326 rpcz_store.cc:276] 0111 02:57:18.973595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.973745 (+ 150us) service_pool.cc:225] Handling call
0111 02:57:20.553915 (+1580170us) ts_tablet_manager.cc:1145] Deleting tablet 94b9d14959df49a6bbd80ba9e9e57ee4
0111 02:57:20.553989 (+ 74us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:20.667920 (+113931us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1575892}
I20250111 02:57:20.669495 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 88) took 1698 ms. Trace:
I20250111 02:57:20.669803 19326 rpcz_store.cc:276] 0111 02:57:18.971335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.982306 (+ 10971us) service_pool.cc:225] Handling call
0111 02:57:20.669144 (+1686838us) ts_tablet_manager.cc:1145] Deleting tablet 2cd364f9a6894d19826deb59a8d8a612
0111 02:57:20.669186 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:20.669438 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1595393}
I20250111 02:57:20.670403 19052 raft_consensus.cc:2463] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:20.670751 19326 tablet_replica.cc:331] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:20.670949 19028 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf1ce65ccf54358b95665b9764fbdb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:20.672377 19326 raft_consensus.cc:2238] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:20.673810 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet 5118d4b9cfbb49e8af6af56180858336 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:20.674927 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet 066c87476a854e169fd4f5f1cdbb49f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:20.679787 18986 leader_election.cc:304] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f; no voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:20.680905 19533 raft_consensus.cc:3054] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:20.682053 18621 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_e7b6efb1d2ad4fefbfa0732b863704eb" } new_table_name: "default.Table_6d5220f0af774036a2c0a13c8eb24728" modify_external_catalogs: true
I20250111 02:57:20.681977 18622 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_ea096f0c1f404b01ad15decac6707122"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:20.683996 18622 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea096f0c1f404b01ad15decac6707122 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:20.686748 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 6a2234755d474740a55184660119eb11 (table default.Table_f05e5d6100bc46379e8598ce5d367e03 [id=9493f23c2c874bda94417f245e37e24f]) successfully deleted
W20250111 02:57:20.687517 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet ef70b417568049f0b496f4d8e5f7e0e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef70b417568049f0b496f4d8e5f7e0e4 already in progress: creating tablet
I20250111 02:57:20.691090 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_aea3cbca3247424c9a12e310135fe0b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:20.691955 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 94b9d14959df49a6bbd80ba9e9e57ee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94b9d14959df49a6bbd80ba9e9e57ee4 already in progress: creating tablet
W20250111 02:57:20.693135 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aea3cbca3247424c9a12e310135fe0b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:20.696786 18627 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 395e619fb45447b59a67afe7cf36ef5a
I20250111 02:57:20.681495 19023 tablet_service.cc:1467] Processing CreateTablet for tablet e4bd42ad66064bcaa043746081f1c759 (DEFAULT_TABLE table=default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.700323 19023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4bd42ad66064bcaa043746081f1c759. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.700855 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 3c02166674a140f29e9f655d09532ced (table default.Table_c2796c2dda744b6c8c8b1b8a7bed2a19 [id=820cc3b845124b1ea333e55aa6537fad]) successfully deleted
I20250111 02:57:20.703135 18882 tablet_service.cc:1467] Processing CreateTablet for tablet 2e06e867266948df8eb35a85c58f578f (DEFAULT_TABLE table=default.Table_4dd6f581bd9c4feea48d8aaf545dee06 [id=b200db65051a40e38aca55f359939a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.704283 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e06e867266948df8eb35a85c58f578f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.705045 19330 raft_consensus.cc:357] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.705802 19330 raft_consensus.cc:383] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:20.706065 19330 raft_consensus.cc:738] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:20.590721 19321 raft_consensus.cc:357] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.707340 19321 raft_consensus.cc:383] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:20.707608 19321 raft_consensus.cc:738] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:20.708216 19321 consensus_queue.cc:260] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.709213 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef86e1fd412484db83e7d39c9059a90 (DEFAULT_TABLE table=default.Table_91137e45d2b549c6a6c6a478c43c1033 [id=c0c4000c9f0642c8924deea7ed88234a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.710371 18878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ef86e1fd412484db83e7d39c9059a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.710618 19321 ts_tablet_manager.cc:1428] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.122s user 0.005s sys 0.001s
I20250111 02:57:20.711817 19321 tablet_bootstrap.cc:492] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:20.688328 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf1ce65ccf54358b95665b9764fbdb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:32868
W20250111 02:57:20.715796 18986 leader_election.cc:343] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:20.716238 18986 leader_election.cc:304] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f; no voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:20.716943 19481 raft_consensus.cc:2743] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:20.689538 19169 tablet_service.cc:1467] Processing CreateTablet for tablet e4bd42ad66064bcaa043746081f1c759 (DEFAULT_TABLE table=default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.719282 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4bd42ad66064bcaa043746081f1c759. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.706681 19330 consensus_queue.cc:260] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.737103 19330 ts_tablet_manager.cc:1428] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.223s user 0.007s sys 0.000s
I20250111 02:57:20.739274 19330 tablet_bootstrap.cc:492] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:20.739699 19326 raft_consensus.cc:2267] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:20.743877 19326 ts_tablet_manager.cc:1905] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:20.745544 19322 tablet_bootstrap.cc:492] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:20.745981 19322 ts_tablet_manager.cc:1397] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.341s user 0.008s sys 0.006s
W20250111 02:57:20.752606 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 2cd364f9a6894d19826deb59a8d8a612 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cd364f9a6894d19826deb59a8d8a612 already in progress: creating tablet
W20250111 02:57:20.756440 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 5cf1ce65ccf54358b95665b9764fbdb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cf1ce65ccf54358b95665b9764fbdb0 already in progress: creating tablet
I20250111 02:57:20.757494 19327 tablet_replica.cc:331] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:20.758586 19327 raft_consensus.cc:2238] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:20.760145 19033 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef86e1fd412484db83e7d39c9059a90 (DEFAULT_TABLE table=default.Table_91137e45d2b549c6a6c6a478c43c1033 [id=c0c4000c9f0642c8924deea7ed88234a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.761675 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ef86e1fd412484db83e7d39c9059a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.769856 19028 tablet_service.cc:1467] Processing CreateTablet for tablet 3052e9707de14e1f9e38236d56908ed3 (DEFAULT_TABLE table=default.Table_23d632d4476c46b78d27619201589641 [id=db5ffbbe4fe7424f943fbaca17103d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.771817 19028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3052e9707de14e1f9e38236d56908ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.772814 19533 raft_consensus.cc:2743] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:20.790464 19131 leader_election.cc:304] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:20.753294 18880 tablet_service.cc:1467] Processing CreateTablet for tablet 3052e9707de14e1f9e38236d56908ed3 (DEFAULT_TABLE table=default.Table_23d632d4476c46b78d27619201589641 [id=db5ffbbe4fe7424f943fbaca17103d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.792260 19327 raft_consensus.cc:2267] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:20.792747 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3052e9707de14e1f9e38236d56908ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.795552 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_7ee7d20cfb7e4985bdd8539d15e354fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:20.797091 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ee7d20cfb7e4985bdd8539d15e354fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:20.704546 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "066c87476a854e169fd4f5f1cdbb49f7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:20.803889 19025 tablet_service.cc:1467] Processing CreateTablet for tablet 2e06e867266948df8eb35a85c58f578f (DEFAULT_TABLE table=default.Table_4dd6f581bd9c4feea48d8aaf545dee06 [id=b200db65051a40e38aca55f359939a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.805512 19025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e06e867266948df8eb35a85c58f578f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.808295 19545 raft_consensus.cc:491] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:20.808763 19545 raft_consensus.cc:513] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:20.809695 19444 raft_consensus.cc:2798] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:20.810163 19444 raft_consensus.cc:695] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:20.811074 19444 consensus_queue.cc:237] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.812306 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1802e7349934fbb9d5f7eda37c3df20" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:20.812964 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1802e7349934fbb9d5f7eda37c3df20" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:20.814253 18842 leader_election.cc:343] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:20.815407 18841 leader_election.cc:343] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:20.815809 18841 leader_election.cc:304] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:20.810828 19545 leader_election.cc:290] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:20.816493 19427 raft_consensus.cc:2743] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:20.817767 19167 tablet_service.cc:1467] Processing CreateTablet for tablet 2e06e867266948df8eb35a85c58f578f (DEFAULT_TABLE table=default.Table_4dd6f581bd9c4feea48d8aaf545dee06 [id=b200db65051a40e38aca55f359939a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.818866 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e06e867266948df8eb35a85c58f578f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.820489 19168 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef86e1fd412484db83e7d39c9059a90 (DEFAULT_TABLE table=default.Table_91137e45d2b549c6a6c6a478c43c1033 [id=c0c4000c9f0642c8924deea7ed88234a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.821604 19168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ef86e1fd412484db83e7d39c9059a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.771893 19322 raft_consensus.cc:357] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.828967 19322 raft_consensus.cc:383] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:20.829694 19322 raft_consensus.cc:738] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:20.835503 19322 consensus_queue.cc:260] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.845899 19322 ts_tablet_manager.cc:1428] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.100s user 0.006s sys 0.000s
I20250111 02:57:20.847708 19322 tablet_bootstrap.cc:492] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:20.862818 19322 tablet_bootstrap.cc:654] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:20.863307 19327 ts_tablet_manager.cc:1905] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:20.891453 19330 tablet_bootstrap.cc:654] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:20.901986 19322 tablet_bootstrap.cc:492] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:20.902436 19322 ts_tablet_manager.cc:1397] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.055s user 0.011s sys 0.004s
I20250111 02:57:20.904891 19322 raft_consensus.cc:357] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.905598 19322 raft_consensus.cc:383] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:20.905844 19322 raft_consensus.cc:738] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:20.906453 19322 consensus_queue.cc:260] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:20.910547 19326 ts_tablet_manager.cc:1918] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:20.912845 19325 ts_tablet_manager.cc:1918] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:20.913396 19325 log.cc:1198] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/b715e32011114e6286c0b8e5406258c4
I20250111 02:57:20.913789 19322 ts_tablet_manager.cc:1428] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.011s user 0.003s sys 0.004s
I20250111 02:57:20.914827 19322 tablet_bootstrap.cc:492] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:20.916983 19325 ts_tablet_manager.cc:1939] T b715e32011114e6286c0b8e5406258c4 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:20.923199 19322 tablet_bootstrap.cc:654] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:20.929039 19321 tablet_bootstrap.cc:654] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:20.945608 18841 leader_election.cc:304] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:20.946849 19545 raft_consensus.cc:2743] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250111 02:57:20.922698 19171 tablet_service.cc:1467] Processing CreateTablet for tablet 3052e9707de14e1f9e38236d56908ed3 (DEFAULT_TABLE table=default.Table_23d632d4476c46b78d27619201589641 [id=db5ffbbe4fe7424f943fbaca17103d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:20.948881 19171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3052e9707de14e1f9e38236d56908ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:20.946614 18627 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet b5148fa2e6504561bd1b1a19e48993d7 (table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0]). Expected version 1 got 0
I20250111 02:57:20.910900 19326 log.cc:1198] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/adf93d013fb84ada9ff86ad964273e00
I20250111 02:57:20.976444 19326 ts_tablet_manager.cc:1939] T adf93d013fb84ada9ff86ad964273e00 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:20.979291 18629 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_e2df5efa4e624d0fa2c7c164dd44f826" } new_table_name: "default.Table_62556c9dbbcb4e8a83e01aef82e41449" modify_external_catalogs: true
I20250111 02:57:20.979615 18626 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_4dd6f581bd9c4feea48d8aaf545dee06" } modify_external_catalogs: true
I20250111 02:57:20.980544 18626 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_4dd6f581bd9c4feea48d8aaf545dee06" } modify_external_catalogs: true
I20250111 02:57:20.982928 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_d693cd14e5074b6b8ef78428cb948e48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:20.986308 18623 master_service.cc:946] ReplaceTablet: received request to replace tablet 5118d4b9cfbb49e8af6af56180858336 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:20.987161 19533 raft_consensus.cc:491] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250111 02:57:20.987342 18621 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ef92ac716824107916a1ea4fd49d5aa has no consensus state
I20250111 02:57:20.975703 18627 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet 5baef8688ede42fca28cfb6b0bdeefa6 (table default.Table_7f4578da9f5e418c9d67a535471b45cd [id=154c37dc0dba44bdab5e0315756dc7cf]). Expected version 1 got 0
I20250111 02:57:20.987614 19533 raft_consensus.cc:513] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:20.988011 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d693cd14e5074b6b8ef78428cb948e48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:20.988772 18621 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_7f4578da9f5e418c9d67a535471b45cd" } modify_external_catalogs: true
I20250111 02:57:20.989475 18621 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_7f4578da9f5e418c9d67a535471b45cd" } modify_external_catalogs: true
I20250111 02:57:20.989599 19533 leader_election.cc:290] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:20.991307 18623 master_service.cc:946] ReplaceTablet: received request to replace tablet 4b302fcba27244c193586bc25fc9305f from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:20.991245 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54311f4269d4575b8998d7f545000e7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:20.993716 18986 leader_election.cc:343] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:20.994771 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54311f4269d4575b8998d7f545000e7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:20.996309 18989 leader_election.cc:343] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:20.996778 18989 leader_election.cc:304] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f; no voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:20.998116 19533 raft_consensus.cc:2743] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:21.004017 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 97) took 1612 ms. Trace:
I20250111 02:57:20.994524 18625 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet 5baef8688ede42fca28cfb6b0bdeefa6 (table default.Table_7f4578da9f5e418c9d67a535471b45cd [id=154c37dc0dba44bdab5e0315756dc7cf]). Expected version 1 got 0
I20250111 02:57:21.014547 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 85) took 2776 ms. Trace:
I20250111 02:57:21.015734 19326 rpcz_store.cc:276] 0111 02:57:19.391983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:19.392165 (+ 182us) service_pool.cc:225] Handling call
0111 02:57:20.670610 (+1278445us) ts_tablet_manager.cc:1145] Deleting tablet adf93d013fb84ada9ff86ad964273e00
0111 02:57:20.670649 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:20.785607 (+114958us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:20.910324 (+124717us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:21.003934 (+ 93610us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1277185}
I20250111 02:57:21.019105 19330 tablet_bootstrap.cc:492] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:21.019595 19330 ts_tablet_manager.cc:1397] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.281s user 0.016s sys 0.003s
I20250111 02:57:21.020370 19326 tablet_replica.cc:331] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:21.021744 19326 raft_consensus.cc:2238] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:21.022275 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet adf93d013fb84ada9ff86ad964273e00 (table default.Table_baed23f14d1d4ebb978e45731f534aba [id=ec872ce86d1940cb85adae5f3047321d]) successfully deleted
I20250111 02:57:21.028293 19326 raft_consensus.cc:2267] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:21.030207 19326 ts_tablet_manager.cc:1905] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:21.037243 19325 rpcz_store.cc:276] 0111 02:57:18.238402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.238558 (+ 156us) service_pool.cc:225] Handling call
0111 02:57:20.605058 (+2366500us) ts_tablet_manager.cc:1145] Deleting tablet b715e32011114e6286c0b8e5406258c4
0111 02:57:20.605138 (+ 80us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:20.780647 (+175509us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:20.912611 (+131964us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:21.012848 (+100237us) spinlock_profiling.cc:243] Waited 93.2 ms on lock 0x7b5800001898. stack: 00007fe197db8853 00007fe197db8750 00007fe19792d584 00000000004fcd24 00000000004fccb9 00007fe197c2e2d1 00007fe197dad291 00007fe197dad8c2 00007fe1a0a3f999 00007fe19fca45f8 00007fe19fd68ea2 00007fe19fd98ca0 00007fe19fd7fe6d 00007fe19fd7fe19 00007fe19fd7fda9 00007fe19fd7fd71
0111 02:57:21.014478 (+ 1630us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":214388608,"tablet-delete.queue_time_us":2263554}
W20250111 02:57:21.038506 18629 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0267a92995ea460786dc26813b5895cf consensus state has no leader
I20250111 02:57:21.041954 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet b715e32011114e6286c0b8e5406258c4 (table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]) successfully deleted
I20250111 02:57:21.029306 19321 tablet_bootstrap.cc:492] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:21.022138 19330 raft_consensus.cc:357] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.043738 19321 ts_tablet_manager.cc:1397] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.332s user 0.012s sys 0.003s
I20250111 02:57:21.043676 19330 raft_consensus.cc:383] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:21.043983 19330 raft_consensus.cc:738] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:21.044629 19330 consensus_queue.cc:260] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.046659 19330 ts_tablet_manager.cc:1428] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.027s user 0.004s sys 0.000s
I20250111 02:57:21.051173 19321 raft_consensus.cc:357] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.051843 19322 tablet_bootstrap.cc:492] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:21.051882 19321 raft_consensus.cc:383] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:21.052476 19321 raft_consensus.cc:738] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:21.053210 19321 consensus_queue.cc:260] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.058408 19330 tablet_bootstrap.cc:492] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:21.057740 19322 ts_tablet_manager.cc:1397] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.143s user 0.023s sys 0.025s
I20250111 02:57:21.061890 19533 raft_consensus.cc:491] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:21.062387 19533 raft_consensus.cc:513] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.064253 19533 leader_election.cc:290] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:21.064138 19322 raft_consensus.cc:357] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.064841 19322 raft_consensus.cc:383] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:21.065114 19322 raft_consensus.cc:738] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:21.065786 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8128b3941d54ce2a72bf2ae762a774f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:21.066416 18909 raft_consensus.cc:2463] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:21.067951 18989 leader_election.cc:304] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:21.069473 19533 raft_consensus.cc:2798] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:21.069784 19533 raft_consensus.cc:491] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:21.070111 19533 raft_consensus.cc:3054] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:21.078178 19322 consensus_queue.cc:260] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.084195 19330 tablet_bootstrap.cc:654] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:21.086457 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8128b3941d54ce2a72bf2ae762a774f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:21.087672 19322 ts_tablet_manager.cc:1428] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
W20250111 02:57:21.088027 18986 leader_election.cc:343] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:21.095558 19533 raft_consensus.cc:513] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.098716 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8128b3941d54ce2a72bf2ae762a774f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:21.099357 18909 raft_consensus.cc:3054] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:21.094848 18630 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet b5148fa2e6504561bd1b1a19e48993d7 (table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0]). Expected version 1 got 0
I20250111 02:57:21.107183 19533 leader_election.cc:290] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:21.107133 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8128b3941d54ce2a72bf2ae762a774f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
W20250111 02:57:21.109031 18986 leader_election.cc:343] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:21.108762 18909 raft_consensus.cc:2463] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:21.110585 19322 tablet_bootstrap.cc:492] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:21.110479 18989 leader_election.cc:304] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters: d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:21.112242 19321 ts_tablet_manager.cc:1428] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.068s user 0.006s sys 0.000s
I20250111 02:57:21.120000 18621 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 5baef8688ede42fca28cfb6b0bdeefa6
I20250111 02:57:21.121356 19321 tablet_bootstrap.cc:492] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:21.123085 19325 tablet_replica.cc:331] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:21.123780 19325 raft_consensus.cc:2238] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:21.124305 19325 raft_consensus.cc:2267] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:21.126228 19325 ts_tablet_manager.cc:1905] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:21.129626 19481 raft_consensus.cc:2798] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:21.130733 19322 tablet_bootstrap.cc:654] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:21.133674 19326 ts_tablet_manager.cc:1918] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:21.143302 19326 log.cc:1198] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/72bcbe140a82462d909bd6d41bfbd2af
I20250111 02:57:21.144209 19326 ts_tablet_manager.cc:1939] T 72bcbe140a82462d909bd6d41bfbd2af P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:21.147430 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet 5baef8688ede42fca28cfb6b0bdeefa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:21.151609 19533 raft_consensus.cc:491] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:21.151819 19322 tablet_bootstrap.cc:492] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:21.152114 19533 raft_consensus.cc:513] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.153223 19322 ts_tablet_manager.cc:1397] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.000s
I20250111 02:57:21.155171 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe4e60db4244a9a9696153b630c575b" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:21.155506 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe4e60db4244a9a9696153b630c575b" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:21.155929 18909 raft_consensus.cc:2463] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:21.156003 19322 raft_consensus.cc:357] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.156694 19322 raft_consensus.cc:383] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:21.156962 19322 raft_consensus.cc:738] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
W20250111 02:57:21.156790 18615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.124:34593, user_credentials={real_user=slave}} blocked reactor thread for 44911.1us
I20250111 02:57:21.156980 18989 leader_election.cc:304] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
W20250111 02:57:21.157586 18986 leader_election.cc:343] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:21.161127 19322 consensus_queue.cc:260] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.162626 19025 tablet_service.cc:1514] Processing DeleteTablet for tablet ede92343d9b54c29b8173a5eafcf68c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 199ca8692f39445f93d8c27e2134a32c) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:21.163631 19025 tablet_service.cc:1514] Processing DeleteTablet for tablet 616655aa75824fffa46f81c4f95d5159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:21.165019 19025 tablet_service.cc:1514] Processing DeleteTablet for tablet b54311f4269d4575b8998d7f545000e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 845da3ee0863430ea3fe9de8c84f2c17) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:21.166913 19025 tablet_service.cc:1514] Processing DeleteTablet for tablet d8128b3941d54ce2a72bf2ae762a774f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:21.171495 19322 ts_tablet_manager.cc:1428] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20250111 02:57:21.176666 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet 5baef8688ede42fca28cfb6b0bdeefa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:21.180909 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe4e60db4244a9a9696153b630c575b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:21.181671 19562 raft_consensus.cc:2798] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:21.185525 19322 tablet_bootstrap.cc:492] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:21.185891 19562 raft_consensus.cc:491] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:21.185864 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet d8128b3941d54ce2a72bf2ae762a774f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:21.186846 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 5baef8688ede42fca28cfb6b0bdeefa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:21.190472 19562 raft_consensus.cc:3054] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:21.188458 19533 leader_election.cc:290] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:21.188143 19330 tablet_bootstrap.cc:492] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:21.191428 19330 ts_tablet_manager.cc:1397] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.133s user 0.010s sys 0.008s
I20250111 02:57:21.191402 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet 29561e984e6d40fea032075f0531dd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:21.194008 19330 raft_consensus.cc:357] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:21.197170 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet 29561e984e6d40fea032075f0531dd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:21.194903 19330 raft_consensus.cc:383] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:21.199401 19330 raft_consensus.cc:738] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:21.200120 19330 consensus_queue.cc:260] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:21.202306 19330 ts_tablet_manager.cc:1428] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250111 02:57:21.147579 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 96) took 1756 ms. Trace:
I20250111 02:57:21.203289 19326 rpcz_store.cc:276] 0111 02:57:19.391687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:19.393366 (+ 1679us) service_pool.cc:225] Handling call
0111 02:57:21.019702 (+1626336us) ts_tablet_manager.cc:1145] Deleting tablet 72bcbe140a82462d909bd6d41bfbd2af
0111 02:57:21.019742 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:21.065558 (+ 45816us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:21.133453 (+ 67895us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:21.147512 (+ 14059us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1625769}
I20250111 02:57:21.204730 19326 tablet_replica.cc:331] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:21.205538 19326 raft_consensus.cc:2238] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:21.206082 19326 raft_consensus.cc:2267] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:21.214083 19321 tablet_bootstrap.cc:654] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:21.219866 19562 raft_consensus.cc:513] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:14.814064 19375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:57:21.223192 19374 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 6413 milliseconds
I20250111 02:57:21.225085 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 72bcbe140a82462d909bd6d41bfbd2af (table default.Table_8d238bdc5f53451aab66467655c6fb2b [id=f35e4781910c45ba93c35a24da2b83c1]) successfully deleted
I20250111 02:57:21.225726 18627 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_ea096f0c1f404b01ad15decac6707122" } new_table_name: "default.Table_0928c84124ee478d802678dfddd36317" modify_external_catalogs: true
I20250111 02:57:21.226274 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe4e60db4244a9a9696153b630c575b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:36190
W20250111 02:57:14.815696 19370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:21.224064 19309 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250111 02:57:21.236094 19330 tablet_bootstrap.cc:492] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:21.237738 19309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:57:21.238835 19322 tablet_bootstrap.cc:654] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:21.242313 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet b54311f4269d4575b8998d7f545000e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 845da3ee0863430ea3fe9de8c84f2c17) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:21.243177 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe4e60db4244a9a9696153b630c575b" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
W20250111 02:57:21.244840 18986 leader_election.cc:343] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:21.246245 19325 ts_tablet_manager.cc:1918] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:21.246680 19325 log.cc:1198] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/8bfad6c08b4b44c1a664cda3b43acbb1
I20250111 02:57:21.247555 19325 ts_tablet_manager.cc:1939] T 8bfad6c08b4b44c1a664cda3b43acbb1 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:21.243379 19562 leader_election.cc:290] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:21.249890 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 87) took 2834 ms. Trace:
I20250111 02:57:21.250643 19481 raft_consensus.cc:695] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:21.252132 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe4e60db4244a9a9696153b630c575b" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:21.252496 19327 ts_tablet_manager.cc:1918] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:21.252732 18908 raft_consensus.cc:3054] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:21.253232 19481 consensus_queue.cc:237] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.261034 19322 tablet_bootstrap.cc:492] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:21.261566 19322 ts_tablet_manager.cc:1397] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.076s user 0.013s sys 0.002s
I20250111 02:57:21.264173 18908 raft_consensus.cc:2463] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:21.264292 19322 raft_consensus.cc:357] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.265080 19322 raft_consensus.cc:383] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:21.265375 19322 raft_consensus.cc:738] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:21.270367 19322 consensus_queue.cc:260] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.272322 19309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:57:21.277406 19562 consensus_queue.cc:1035] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:21.282810 19330 tablet_bootstrap.cc:654] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:21.284996 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_24e5a7f819cc45dbb7efbd7286ad7005"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:21.286608 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24e5a7f819cc45dbb7efbd7286ad7005 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:21.290614 18624 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_eaa40a643c0e43b9a1204616e7403efa" } new_table_name: "default.Table_b44f1ebbaf7045e2bcfefcee1c91c0d9" modify_external_catalogs: true
I20250111 02:57:21.296653 19321 tablet_bootstrap.cc:492] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:21.297137 19321 ts_tablet_manager.cc:1397] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.176s user 0.010s sys 0.000s
I20250111 02:57:21.300377 19321 raft_consensus.cc:357] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:21.300355 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_3e9cad78bc1447c8806da38d51f85778"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:21.301223 19309 hybrid_clock.cc:648] HybridClock initialized: now 1736564241300783 us; error 432 us; skew 500 ppm
W20250111 02:57:21.302081 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e9cad78bc1447c8806da38d51f85778 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:21.302476 19309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:57:21.250238 19325 rpcz_store.cc:276] 0111 02:57:18.415536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.415674 (+ 138us) service_pool.cc:225] Handling call
0111 02:57:21.043336 (+2627662us) ts_tablet_manager.cc:1145] Deleting tablet 8bfad6c08b4b44c1a664cda3b43acbb1
0111 02:57:21.122931 (+ 79595us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:21.141229 (+ 18298us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:21.245968 (+104739us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:21.249821 (+ 3853us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2577819}
I20250111 02:57:21.303983 18621 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet b8e8c72c99ac4c369957b571d74baffe (table default.Table_35d7f864912b40bcb5251a90d40c7849 [id=772d994bee2d4e34bd48912acecf8d48]). Expected version 1 got 0
I20250111 02:57:21.304667 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 8bfad6c08b4b44c1a664cda3b43acbb1 (table default.Table_65af1591521e4974a28304ce51fe6fb1 [id=7cf2761276494576a4c2be6239374334]) successfully deleted
I20250111 02:57:21.304841 18621 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet b5148fa2e6504561bd1b1a19e48993d7 (table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0]). Expected version 1 got 0
I20250111 02:57:21.301015 19321 raft_consensus.cc:383] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:21.308434 19321 raft_consensus.cc:738] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:21.309834 19321 consensus_queue.cc:260] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:21.312815 19168 tablet_service.cc:1467] Processing CreateTablet for tablet 061e6eb5775443679e9cbd6b11f66fe3 (DEFAULT_TABLE table=default.Table_c398ec562f594aca8c1e62cde38c04f9 [id=14bc02cbe30c486782d8fa4396855a9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.312992 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 061e6eb5775443679e9cbd6b11f66fe3 (DEFAULT_TABLE table=default.Table_c398ec562f594aca8c1e62cde38c04f9 [id=14bc02cbe30c486782d8fa4396855a9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.314350 19168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 061e6eb5775443679e9cbd6b11f66fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.320367 19321 ts_tablet_manager.cc:1428] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20250111 02:57:21.291970 18989 leader_election.cc:304] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters: d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:21.324688 18890 tablet_service.cc:1467] Processing CreateTablet for tablet 2f11541b436e42b7b0728da2aaec48f7 (DEFAULT_TABLE table=default.Table_f36240205653470e9a67400b787ce860 [id=69f6e7b3ec40423a9d0bef5487ffe84a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.325659 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet f1802e7349934fbb9d5f7eda37c3df20 (table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0]). Expected version 4 got 0
I20250111 02:57:21.327273 19562 raft_consensus.cc:2798] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:21.327816 19562 raft_consensus.cc:695] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:21.328756 19167 tablet_service.cc:1467] Processing CreateTablet for tablet 2f11541b436e42b7b0728da2aaec48f7 (DEFAULT_TABLE table=default.Table_f36240205653470e9a67400b787ce860 [id=69f6e7b3ec40423a9d0bef5487ffe84a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.328776 19562 consensus_queue.cc:237] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.331522 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f11541b436e42b7b0728da2aaec48f7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:21.333776 18989 consensus_peers.cc:487] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:21.334486 18882 tablet_service.cc:1467] Processing CreateTablet for tablet cb96635f30b34de48870daf5166813bc (DEFAULT_TABLE table=default.Table_39f0114c91ef4d1c86ef0cf522e31e11 [id=5189b95a9b0e4b47ac0d7768e08e902b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.335814 19171 tablet_service.cc:1514] Processing DeleteTablet for tablet 616655aa75824fffa46f81c4f95d5159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:21.339377 19321 tablet_bootstrap.cc:492] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:21.342087 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb96635f30b34de48870daf5166813bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.344063 19171 tablet_service.cc:1467] Processing CreateTablet for tablet cb96635f30b34de48870daf5166813bc (DEFAULT_TABLE table=default.Table_39f0114c91ef4d1c86ef0cf522e31e11 [id=5189b95a9b0e4b47ac0d7768e08e902b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.347608 18890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f11541b436e42b7b0728da2aaec48f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.347743 19171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb96635f30b34de48870daf5166813bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.351527 18878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 061e6eb5775443679e9cbd6b11f66fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.363595 19321 tablet_bootstrap.cc:654] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:21.363889 19327 log.cc:1198] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/5118d4b9cfbb49e8af6af56180858336
I20250111 02:57:21.364580 19327 ts_tablet_manager.cc:1939] T 5118d4b9cfbb49e8af6af56180858336 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:21.366106 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 5118d4b9cfbb49e8af6af56180858336 (table default.Table_39701f385f884aac9233a842fcaf93be [id=bb83163ef256406da8650e72eadf7f6c]) successfully deleted
I20250111 02:57:21.327524 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 92) took 2366 ms. Trace:
I20250111 02:57:21.371348 19325 rpcz_store.cc:276] 0111 02:57:18.961294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.961449 (+ 155us) service_pool.cc:225] Handling call
0111 02:57:21.324227 (+2362778us) ts_tablet_manager.cc:1145] Deleting tablet 2cd364f9a6894d19826deb59a8d8a612
0111 02:57:21.327123 (+ 2896us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:21.327464 (+ 341us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2362062}
I20250111 02:57:21.373183 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 93) took 2406 ms. Trace:
I20250111 02:57:21.373605 19325 rpcz_store.cc:276] 0111 02:57:18.966551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.966755 (+ 204us) service_pool.cc:225] Handling call
0111 02:57:21.372576 (+2405821us) ts_tablet_manager.cc:1145] Deleting tablet 9fe4e60db4244a9a9696153b630c575b
0111 02:57:21.372655 (+ 79us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:21.373111 (+ 456us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2405073}
W20250111 02:57:21.375564 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 2cd364f9a6894d19826deb59a8d8a612 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cd364f9a6894d19826deb59a8d8a612 already in progress: creating tablet
W20250111 02:57:21.376363 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 9fe4e60db4244a9a9696153b630c575b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fe4e60db4244a9a9696153b630c575b already in progress: creating tablet
I20250111 02:57:21.377561 19033 tablet_service.cc:1467] Processing CreateTablet for tablet 061e6eb5775443679e9cbd6b11f66fe3 (DEFAULT_TABLE table=default.Table_c398ec562f594aca8c1e62cde38c04f9 [id=14bc02cbe30c486782d8fa4396855a9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.378583 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 061e6eb5775443679e9cbd6b11f66fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.385051 19025 tablet_service.cc:1467] Processing CreateTablet for tablet 2f11541b436e42b7b0728da2aaec48f7 (DEFAULT_TABLE table=default.Table_f36240205653470e9a67400b787ce860 [id=69f6e7b3ec40423a9d0bef5487ffe84a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.392022 19322 ts_tablet_manager.cc:1428] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.130s user 0.007s sys 0.000s
I20250111 02:57:21.396519 19321 tablet_bootstrap.cc:492] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:21.396992 19321 ts_tablet_manager.cc:1397] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.058s user 0.014s sys 0.001s
I20250111 02:57:21.397421 19322 tablet_bootstrap.cc:492] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:21.374941 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 94) took 2400 ms. Trace:
I20250111 02:57:21.399420 19325 rpcz_store.cc:276] 0111 02:57:18.974698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:18.974840 (+ 142us) service_pool.cc:225] Handling call
0111 02:57:21.374469 (+2399629us) ts_tablet_manager.cc:1145] Deleting tablet 94b9d14959df49a6bbd80ba9e9e57ee4
0111 02:57:21.374540 (+ 71us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:21.374878 (+ 338us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2398802}
I20250111 02:57:21.399909 19321 raft_consensus.cc:357] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.400645 19321 raft_consensus.cc:383] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250111 02:57:21.400647 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 94b9d14959df49a6bbd80ba9e9e57ee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94b9d14959df49a6bbd80ba9e9e57ee4 already in progress: creating tablet
I20250111 02:57:21.400923 19321 raft_consensus.cc:738] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:21.401116 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 99) took 2035 ms. Trace:
I20250111 02:57:21.401422 19325 rpcz_store.cc:276] 0111 02:57:19.365324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:19.365481 (+ 157us) service_pool.cc:225] Handling call
0111 02:57:21.400658 (+2035177us) ts_tablet_manager.cc:1145] Deleting tablet 92aa64b2de5746d1b12cc93a10c9d56e
0111 02:57:21.400729 (+ 71us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:21.401065 (+ 336us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2034494}
W20250111 02:57:21.402640 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 92aa64b2de5746d1b12cc93a10c9d56e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92aa64b2de5746d1b12cc93a10c9d56e already in progress: creating tablet
I20250111 02:57:21.405208 18886 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef92ac716824107916a1ea4fd49d5aa (DEFAULT_TABLE table=default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.409786 18880 tablet_service.cc:1467] Processing CreateTablet for tablet 0b6687f39d4246b49c0c3eb904b11c1a (DEFAULT_TABLE table=default.Table_857f04d0de7e47a4828db9a8997f1009 [id=9cf908e0c7d44b60b88a32cf2d6368a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.410578 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef92ac716824107916a1ea4fd49d5aa (DEFAULT_TABLE table=default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.412039 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ef92ac716824107916a1ea4fd49d5aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.355537 19326 ts_tablet_manager.cc:1905] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:21.412989 19321 consensus_queue.cc:260] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.414250 18629 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_03f0b101f0c64274a40b43897fe6dcdb" } modify_external_catalogs: true
I20250111 02:57:21.414942 18629 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_03f0b101f0c64274a40b43897fe6dcdb" } modify_external_catalogs: true
W20250111 02:57:21.418272 18627 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2873325cece64db6a673d4fe08eb7409 has no consensus state
I20250111 02:57:21.419629 18622 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_e42719b273bf4b8f96fa48d1a3d6c8d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:21.421183 18622 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e42719b273bf4b8f96fa48d1a3d6c8d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:21.421921 19321 ts_tablet_manager.cc:1428] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.025s user 0.005s sys 0.002s
I20250111 02:57:21.422873 19321 tablet_bootstrap.cc:492] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:21.428615 18621 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_fc5930652905494584fde76fff8629d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:21.430650 18621 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc5930652905494584fde76fff8629d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:21.436887 19321 tablet_bootstrap.cc:654] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:21.444386 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 109) took 1804 ms. Trace:
I20250111 02:57:21.444873 19325 rpcz_store.cc:276] 0111 02:57:19.640127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:19.640265 (+ 138us) service_pool.cc:225] Handling call
0111 02:57:21.443761 (+1803496us) ts_tablet_manager.cc:1145] Deleting tablet 32afc2f158254811836fcf00f7f409ed
0111 02:57:21.443857 (+ 96us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:21.444306 (+ 449us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1802764}
I20250111 02:57:21.452199 18884 tablet_service.cc:1514] Processing DeleteTablet for tablet c28958a1018344a48d18e6c0f03cd0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:17 UTC) from {username='slave'} at 127.0.0.1:51082
W20250111 02:57:21.455014 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 32afc2f158254811836fcf00f7f409ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32afc2f158254811836fcf00f7f409ed already in progress: creating tablet
I20250111 02:57:21.386937 19025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f11541b436e42b7b0728da2aaec48f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.459312 18884 tablet_service.cc:1467] Processing CreateTablet for tablet adf61f6cb63d4e8ea46b440e7ecce00d (DEFAULT_TABLE table=default.Table_c0ddb0fbf2f347c0b82acf9060722ed2 [id=964e4311f5ed4578a6f7285a9d2bbafa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.462299 19321 tablet_bootstrap.cc:492] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:21.462738 19321 ts_tablet_manager.cc:1397] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.004s
I20250111 02:57:21.465533 19023 tablet_service.cc:1467] Processing CreateTablet for tablet cb96635f30b34de48870daf5166813bc (DEFAULT_TABLE table=default.Table_39f0114c91ef4d1c86ef0cf522e31e11 [id=5189b95a9b0e4b47ac0d7768e08e902b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.465729 19321 raft_consensus.cc:357] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.466384 19321 raft_consensus.cc:383] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:21.466634 19321 raft_consensus.cc:738] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:21.365818 19327 tablet_replica.cc:331] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:21.469609 19028 tablet_service.cc:1467] Processing CreateTablet for tablet 0b6687f39d4246b49c0c3eb904b11c1a (DEFAULT_TABLE table=default.Table_857f04d0de7e47a4828db9a8997f1009 [id=9cf908e0c7d44b60b88a32cf2d6368a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.467039 19023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb96635f30b34de48870daf5166813bc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:21.477922 18617 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 2873325cece64db6a673d4fe08eb7409 (table default.Table_0928c84124ee478d802678dfddd36317 [id=2d5d8f8a7ade443f91a7da34add25a91], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2873325cece64db6a673d4fe08eb7409 has no consensus state
I20250111 02:57:21.481817 19028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b6687f39d4246b49c0c3eb904b11c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.490379 18886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ef92ac716824107916a1ea4fd49d5aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.491312 19322 tablet_bootstrap.cc:654] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:21.468393 19024 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef92ac716824107916a1ea4fd49d5aa (DEFAULT_TABLE table=default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.493526 19024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ef92ac716824107916a1ea4fd49d5aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.495679 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b6687f39d4246b49c0c3eb904b11c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.498664 19029 tablet_service.cc:1467] Processing CreateTablet for tablet adf61f6cb63d4e8ea46b440e7ecce00d (DEFAULT_TABLE table=default.Table_c0ddb0fbf2f347c0b82acf9060722ed2 [id=964e4311f5ed4578a6f7285a9d2bbafa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.499758 18884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf61f6cb63d4e8ea46b440e7ecce00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.500169 19029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf61f6cb63d4e8ea46b440e7ecce00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.500490 18881 tablet_service.cc:1467] Processing CreateTablet for tablet b26e5db463e94d919640a07a2f8574a8 (DEFAULT_TABLE table=default.Table_75e5f05a97b14bf883909660868761fd [id=33afe1cfccda4b79b1aaf539f56f27f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.501907 18881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b26e5db463e94d919640a07a2f8574a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.510821 19026 tablet_service.cc:1467] Processing CreateTablet for tablet b26e5db463e94d919640a07a2f8574a8 (DEFAULT_TABLE table=default.Table_75e5f05a97b14bf883909660868761fd [id=33afe1cfccda4b79b1aaf539f56f27f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.512272 18879 tablet_service.cc:1467] Processing CreateTablet for tablet b4273afc264843029ed0baf1fc5c8a3e (DEFAULT_TABLE table=default.Table_69d080492ff2442ea30d95edf1c27459 [id=73c59c0d9864497fbc9f19e265d8283f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.512610 19026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b26e5db463e94d919640a07a2f8574a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.513495 18879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4273afc264843029ed0baf1fc5c8a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.533479 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 112) took 1039 ms. Trace:
I20250111 02:57:21.533982 19325 rpcz_store.cc:276] 0111 02:57:20.493796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:20.494008 (+ 212us) service_pool.cc:225] Handling call
0111 02:57:21.532975 (+1038967us) ts_tablet_manager.cc:1145] Deleting tablet 0dbb63ab78f84995ae6d9fb2aae78dcf
0111 02:57:21.533043 (+ 68us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:21.533396 (+ 353us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":969671}
I20250111 02:57:21.540078 19444 raft_consensus.cc:491] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:21.540405 19545 raft_consensus.cc:491] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:21.540482 19169 tablet_service.cc:1467] Processing CreateTablet for tablet adf61f6cb63d4e8ea46b440e7ecce00d (DEFAULT_TABLE table=default.Table_c0ddb0fbf2f347c0b82acf9060722ed2 [id=964e4311f5ed4578a6f7285a9d2bbafa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.540663 19444 raft_consensus.cc:513] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.540935 19545 raft_consensus.cc:513] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.541527 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf61f6cb63d4e8ea46b440e7ecce00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.544418 19545 leader_election.cc:290] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:21.546502 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28958a1018344a48d18e6c0f03cd0a3" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:21.548584 18841 leader_election.cc:343] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:21.548949 19444 leader_election.cc:290] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:21.551429 19321 consensus_queue.cc:260] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.552512 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e8c72c99ac4c369957b571d74baffe" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:21.553261 18908 raft_consensus.cc:2388] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0cedab82b154d62b78809edc6f28bfa in current term 1: Already voted for candidate 8221604733fe4fc0a1fc32de8d6af71c in this term.
I20250111 02:57:21.553905 19321 ts_tablet_manager.cc:1428] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.091s user 0.005s sys 0.000s
I20250111 02:57:21.554504 19325 tablet_replica.cc:331] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:21.555814 19327 raft_consensus.cc:2238] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:21.556452 19327 raft_consensus.cc:2267] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
W20250111 02:57:21.556644 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 0dbb63ab78f84995ae6d9fb2aae78dcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dbb63ab78f84995ae6d9fb2aae78dcf already in progress: creating tablet
I20250111 02:57:21.557842 19321 tablet_bootstrap.cc:492] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:21.558269 19327 ts_tablet_manager.cc:1905] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:21.560294 19325 raft_consensus.cc:2238] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:21.560822 19325 raft_consensus.cc:2267] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:21.567761 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e8c72c99ac4c369957b571d74baffe" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:21.568629 19052 raft_consensus.cc:2388] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0cedab82b154d62b78809edc6f28bfa in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:21.569753 19131 leader_election.cc:304] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:21.572479 19330 tablet_bootstrap.cc:492] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:21.572976 19330 ts_tablet_manager.cc:1397] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.337s user 0.007s sys 0.007s
I20250111 02:57:21.573781 19444 raft_consensus.cc:3054] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:21.574319 19176 tablet_service.cc:1467] Processing CreateTablet for tablet b26e5db463e94d919640a07a2f8574a8 (DEFAULT_TABLE table=default.Table_75e5f05a97b14bf883909660868761fd [id=33afe1cfccda4b79b1aaf539f56f27f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.575377 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b26e5db463e94d919640a07a2f8574a8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:21.580686 18616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.126:45115, user_credentials={real_user=slave}} blocked reactor thread for 47910.6us
I20250111 02:57:21.591635 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28958a1018344a48d18e6c0f03cd0a3" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:21.596365 18616 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af -> Peer 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115): Couldn't send request to peer 1e336d4ddf044bf5b0b6e4f5af634727. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250111 02:57:21.609412 19330 raft_consensus.cc:357] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.610440 19330 raft_consensus.cc:383] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:21.611833 19167 tablet_service.cc:1467] Processing CreateTablet for tablet 0b6687f39d4246b49c0c3eb904b11c1a (DEFAULT_TABLE table=default.Table_857f04d0de7e47a4828db9a8997f1009 [id=9cf908e0c7d44b60b88a32cf2d6368a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.612996 19026 tablet_service.cc:1467] Processing CreateTablet for tablet 933828fd6fb041e9a4889f4ea8a3c5ff (DEFAULT_TABLE table=default.Table_dc22147bd8ba4ea18bc4f41b25b277c0 [id=1d14d647d8c24e31a84ce6ac7e2c217e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.614084 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b6687f39d4246b49c0c3eb904b11c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.614101 19026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 933828fd6fb041e9a4889f4ea8a3c5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.568904 18882 tablet_service.cc:1467] Processing CreateTablet for tablet 933828fd6fb041e9a4889f4ea8a3c5ff (DEFAULT_TABLE table=default.Table_dc22147bd8ba4ea18bc4f41b25b277c0 [id=1d14d647d8c24e31a84ce6ac7e2c217e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250111 02:57:21.594069 18842 leader_election.cc:343] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:21.615036 18842 leader_election.cc:304] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:21.615868 19545 raft_consensus.cc:2743] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:21.619607 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 933828fd6fb041e9a4889f4ea8a3c5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.623270 18627 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_69d080492ff2442ea30d95edf1c27459" } new_table_name: "default.Table_4e3a79a5d2184687a8c51fd2702c5fd0" modify_external_catalogs: true
I20250111 02:57:21.609575 18890 tablet_service.cc:1467] Processing CreateTablet for tablet 934c900baf1b4000af7d7ec60cbb3591 (DEFAULT_TABLE table=default.Table_4f96c1244592444a994b5c5631eaaadc [id=4c817a432cf649799a2589f12a320ebd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.612239 19322 tablet_bootstrap.cc:492] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:21.639281 19322 ts_tablet_manager.cc:1397] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.242s user 0.009s sys 0.004s
I20250111 02:57:21.641712 19322 raft_consensus.cc:357] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.610838 19330 raft_consensus.cc:738] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:21.649274 19322 raft_consensus.cc:383] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:21.649211 19026 tablet_service.cc:1467] Processing CreateTablet for tablet 934c900baf1b4000af7d7ec60cbb3591 (DEFAULT_TABLE table=default.Table_4f96c1244592444a994b5c5631eaaadc [id=4c817a432cf649799a2589f12a320ebd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.649665 19322 raft_consensus.cc:738] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:21.652087 19026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 934c900baf1b4000af7d7ec60cbb3591. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.652848 18630 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet f1802e7349934fbb9d5f7eda37c3df20 (table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0]). Expected version 4 got 0
I20250111 02:57:21.649477 19330 consensus_queue.cc:260] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.650489 19029 tablet_service.cc:1467] Processing CreateTablet for tablet d837bd001aa649e9b3ca94f7cdd9f0fe (DEFAULT_TABLE table=default.Table_b6421db891c148c688d25610559fe16f [id=190cfd157c8d4bd19a8f40cfd80925c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.655856 19309 webserver.cc:458] Webserver started at http://127.18.20.126:36887/ using document root <none> and password file <none>
I20250111 02:57:21.656667 18880 tablet_service.cc:1467] Processing CreateTablet for tablet d837bd001aa649e9b3ca94f7cdd9f0fe (DEFAULT_TABLE table=default.Table_b6421db891c148c688d25610559fe16f [id=190cfd157c8d4bd19a8f40cfd80925c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.648823 18890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 934c900baf1b4000af7d7ec60cbb3591. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.657778 19309 fs_manager.cc:362] Metadata directory not provided
I20250111 02:57:21.658210 19309 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:57:21.660880 19029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d837bd001aa649e9b3ca94f7cdd9f0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.662925 19330 ts_tablet_manager.cc:1428] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.090s user 0.009s sys 0.000s
I20250111 02:57:21.663955 19330 tablet_bootstrap.cc:492] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:21.650527 19322 consensus_queue.cc:260] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.676337 18626 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 2e06e867266948df8eb35a85c58f578f
I20250111 02:57:21.676443 19322 ts_tablet_manager.cc:1428] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.037s user 0.005s sys 0.000s
I20250111 02:57:21.684651 18879 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e06e867266948df8eb35a85c58f578f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:21.685510 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d837bd001aa649e9b3ca94f7cdd9f0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.689026 19562 raft_consensus.cc:491] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:21.689570 19562 raft_consensus.cc:513] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.691985 19562 leader_election.cc:290] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:21.687597 19533 raft_consensus.cc:491] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:21.693871 19545 raft_consensus.cc:491] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:21.693938 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29561e984e6d40fea032075f0531dd1f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:21.693848 19533 raft_consensus.cc:513] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:21.694665 18908 raft_consensus.cc:2463] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:21.694479 19545 raft_consensus.cc:513] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.697253 19533 leader_election.cc:290] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:21.697451 18989 leader_election.cc:304] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:21.699689 19533 raft_consensus.cc:2798] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:21.700892 19533 raft_consensus.cc:491] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:21.701944 19533 raft_consensus.cc:3054] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:21.719115 19024 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e06e867266948df8eb35a85c58f578f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:21.724069 19322 tablet_bootstrap.cc:492] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:21.739908 19330 tablet_bootstrap.cc:654] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:21.755458 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1802e7349934fbb9d5f7eda37c3df20" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:21.756289 18908 raft_consensus.cc:2463] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:21.756736 19545 leader_election.cc:290] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:21.760151 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54311f4269d4575b8998d7f545000e7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:21.760986 19052 raft_consensus.cc:2463] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:21.767648 18989 leader_election.cc:304] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:21.768621 19562 raft_consensus.cc:2798] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:21.768483 19174 tablet_service.cc:1467] Processing CreateTablet for tablet 933828fd6fb041e9a4889f4ea8a3c5ff (DEFAULT_TABLE table=default.Table_dc22147bd8ba4ea18bc4f41b25b277c0 [id=1d14d647d8c24e31a84ce6ac7e2c217e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.768957 19562 raft_consensus.cc:491] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:21.769259 19562 raft_consensus.cc:3054] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:21.769599 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 933828fd6fb041e9a4889f4ea8a3c5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.774885 19330 tablet_bootstrap.cc:492] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:21.775375 19330 ts_tablet_manager.cc:1397] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.112s user 0.009s sys 0.005s
I20250111 02:57:21.778654 19330 raft_consensus.cc:357] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.779527 19330 raft_consensus.cc:383] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:21.779879 19330 raft_consensus.cc:738] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:21.779907 19562 raft_consensus.cc:513] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:21.780819 19330 consensus_queue.cc:260] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.784812 19309 fs_manager.cc:714] Time spent opening directory manager: real 0.045s user 0.004s sys 0.009s
I20250111 02:57:21.785199 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1802e7349934fbb9d5f7eda37c3df20" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:21.786073 18908 raft_consensus.cc:3054] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:21.789592 19325 ts_tablet_manager.cc:1905] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:21.791848 19330 ts_tablet_manager.cc:1428] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.016s user 0.007s sys 0.001s
I20250111 02:57:21.792917 19330 tablet_bootstrap.cc:492] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:21.796036 18908 raft_consensus.cc:2463] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:21.803926 19330 tablet_bootstrap.cc:654] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:21.806414 19321 tablet_bootstrap.cc:654] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:21.809711 19587 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:57:21.811381 19309 fs_manager.cc:730] Time spent opening block manager: real 0.023s user 0.006s sys 0.001s
I20250111 02:57:21.811811 19309 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/data,/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/wal
uuid: "1e336d4ddf044bf5b0b6e4f5af634727"
format_stamp: "Formatted at 2025-01-11 02:57:00 on dist-test-slave-09x0"
I20250111 02:57:21.815944 18842 leader_election.cc:304] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:21.815750 19309 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:57:21.817366 18989 leader_election.cc:304] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:21.818322 19481 raft_consensus.cc:2798] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:21.818771 19481 raft_consensus.cc:695] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:21.819566 19481 consensus_queue.cc:237] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:21.829126 19533 raft_consensus.cc:513] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.832322 19545 raft_consensus.cc:2798] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:21.832930 19562 leader_election.cc:290] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:21.834342 19173 tablet_service.cc:1467] Processing CreateTablet for tablet b4273afc264843029ed0baf1fc5c8a3e (DEFAULT_TABLE table=default.Table_69d080492ff2442ea30d95edf1c27459 [id=73c59c0d9864497fbc9f19e265d8283f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:21.835546 19173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4273afc264843029ed0baf1fc5c8a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:21.839875 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54311f4269d4575b8998d7f545000e7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:21.834142 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29561e984e6d40fea032075f0531dd1f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:21.840538 18908 raft_consensus.cc:3054] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:21.833246 19533 leader_election.cc:290] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:21.834964 19330 tablet_bootstrap.cc:492] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:21.843708 19545 raft_consensus.cc:491] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:21.844017 19330 ts_tablet_manager.cc:1397] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.007s
I20250111 02:57:21.844136 19545 raft_consensus.cc:3054] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:21.846624 19330 raft_consensus.cc:357] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.847456 19330 raft_consensus.cc:383] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:21.847735 19330 raft_consensus.cc:738] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:21.848379 19330 consensus_queue.cc:260] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.839943 19322 tablet_bootstrap.cc:654] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:21.851584 19330 ts_tablet_manager.cc:1428] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250111 02:57:21.852546 19330 tablet_bootstrap.cc:492] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
W20250111 02:57:21.855528 18841 leader_election.cc:343] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:21.863556 19444 raft_consensus.cc:2743] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:21.864509 19444 raft_consensus.cc:491] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:21.865106 19444 raft_consensus.cc:513] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:21.865904 19330 tablet_bootstrap.cc:654] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:21.868909 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1802e7349934fbb9d5f7eda37c3df20" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:21.869946 18909 raft_consensus.cc:2388] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0cedab82b154d62b78809edc6f28bfa in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:21.871348 19444 leader_election.cc:290] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:21.875353 19133 consensus_peers.cc:487] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:21.888288 19330 tablet_bootstrap.cc:492] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:21.888723 19330 ts_tablet_manager.cc:1397] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.000s
I20250111 02:57:21.889454 19321 tablet_bootstrap.cc:492] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:21.889900 19321 ts_tablet_manager.cc:1397] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.332s user 0.005s sys 0.007s
I20250111 02:57:21.891630 19330 raft_consensus.cc:357] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.892382 19330 raft_consensus.cc:383] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:21.892660 19330 raft_consensus.cc:738] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:21.893304 19330 consensus_queue.cc:260] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.897679 19330 ts_tablet_manager.cc:1428] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250111 02:57:21.892726 19321 raft_consensus.cc:357] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.898777 19330 tablet_bootstrap.cc:492] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:21.905540 19322 tablet_bootstrap.cc:492] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:21.905972 19322 ts_tablet_manager.cc:1397] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.182s user 0.013s sys 0.001s
I20250111 02:57:21.899052 19321 raft_consensus.cc:383] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:21.907712 19321 raft_consensus.cc:738] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:21.908458 19321 consensus_queue.cc:260] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.921475 19545 raft_consensus.cc:513] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.925282 19564 raft_consensus.cc:491] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:21.925773 19321 ts_tablet_manager.cc:1428] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20250111 02:57:21.925889 19564 raft_consensus.cc:513] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.926751 19321 tablet_bootstrap.cc:492] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:21.928295 19564 leader_election.cc:290] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:21.930086 19533 raft_consensus.cc:491] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:21.932119 19533 raft_consensus.cc:513] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.934598 19330 tablet_bootstrap.cc:654] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:21.936924 19533 leader_election.cc:290] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:21.939714 19309 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:57:21.905102 18908 raft_consensus.cc:2463] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:21.941437 19322 raft_consensus.cc:357] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.942318 19309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:57:21.942378 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54311f4269d4575b8998d7f545000e7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:21.943039 19309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:57:21.943113 19052 raft_consensus.cc:3054] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:21.943552 18623 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 4b302fcba27244c193586bc25fc9305f
I20250111 02:57:21.945441 18623 catalog_manager.cc:6596] ReplaceTablet: tablet 4b302fcba27244c193586bc25fc9305f deleted and replaced by tablet 13f0a0e7698e49a5a57562dc8ad52cd5
I20250111 02:57:21.948022 19322 raft_consensus.cc:383] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:21.948338 19322 raft_consensus.cc:738] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:21.949234 18989 leader_election.cc:304] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:21.949950 19029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b302fcba27244c193586bc25fc9305f with delete_type TABLET_DATA_DELETED (replaced by tablet 13f0a0e7698e49a5a57562dc8ad52cd5) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:21.952178 19533 raft_consensus.cc:2798] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:21.953153 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54311f4269d4575b8998d7f545000e7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:21.953420 19533 raft_consensus.cc:695] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:21.954056 19195 raft_consensus.cc:3054] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:21.954962 19545 leader_election.cc:290] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:21.954291 19322 consensus_queue.cc:260] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.934746 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28958a1018344a48d18e6c0f03cd0a3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:21.936307 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbb63ab78f84995ae6d9fb2aae78dcf" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:21.959638 18909 raft_consensus.cc:2463] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:21.959776 18910 raft_consensus.cc:2463] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:21.961766 18989 leader_election.cc:304] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:21.963801 19562 raft_consensus.cc:2798] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250111 02:57:21.964509 18624 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea76b354a75041e888cf862250e4947b consensus state has no leader
I20250111 02:57:21.964385 19562 raft_consensus.cc:491] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:21.964888 19562 raft_consensus.cc:3054] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:21.968165 18989 leader_election.cc:304] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:21.969537 19322 ts_tablet_manager.cc:1428] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.063s user 0.007s sys 0.001s
I20250111 02:57:21.971591 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b302fcba27244c193586bc25fc9305f with delete_type TABLET_DATA_DELETED (replaced by tablet 13f0a0e7698e49a5a57562dc8ad52cd5) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:21.975716 19564 raft_consensus.cc:2798] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:21.976303 19564 raft_consensus.cc:491] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:21.976809 19564 raft_consensus.cc:3054] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:21.982856 19322 tablet_bootstrap.cc:492] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:21.986390 19321 tablet_bootstrap.cc:654] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:21.958810 19533 consensus_queue.cc:237] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:21.989987 19033 tablet_service.cc:1467] Processing CreateTablet for tablet b4273afc264843029ed0baf1fc5c8a3e (DEFAULT_TABLE table=default.Table_69d080492ff2442ea30d95edf1c27459 [id=73c59c0d9864497fbc9f19e265d8283f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:22.005288 19322 tablet_bootstrap.cc:654] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:22.007133 18630 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_62556c9dbbcb4e8a83e01aef82e41449" } modify_external_catalogs: true
I20250111 02:57:22.007699 18630 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_62556c9dbbcb4e8a83e01aef82e41449" } modify_external_catalogs: true
I20250111 02:57:22.007833 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_5e21879e67544c71af627f795fd6d940"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:22.009442 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e21879e67544c71af627f795fd6d940 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:22.023646 19052 raft_consensus.cc:2463] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:22.025661 18842 leader_election.cc:304] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:22.027231 19545 raft_consensus.cc:2798] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:22.031505 18628 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet b5148fa2e6504561bd1b1a19e48993d7 (table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0]). Expected version 1 got 0
I20250111 02:57:22.035044 19564 raft_consensus.cc:513] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.043344 19562 raft_consensus.cc:513] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.049067 18880 tablet_service.cc:1514] Processing DeleteTablet for tablet ef70b417568049f0b496f4d8e5f7e0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7ea78feee64340e39404cba0f1bf04db) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:22.055039 19533 raft_consensus.cc:491] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:22.055750 19533 raft_consensus.cc:513] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.057354 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dbb63ab78f84995ae6d9fb2aae78dcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:22.059664 19545 raft_consensus.cc:695] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 LEADER]: Becoming Leader. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Running, Role: LEADER
I20250111 02:57:22.061158 19545 consensus_queue.cc:237] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.068861 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ef3f45e5784a29b3b32cc0de6bac84" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:22.070691 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4273afc264843029ed0baf1fc5c8a3e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:22.071101 18989 leader_election.cc:343] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:22.071870 19533 leader_election.cc:290] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:22.059382 19330 tablet_bootstrap.cc:492] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:22.072687 19330 ts_tablet_manager.cc:1397] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.174s user 0.003s sys 0.014s
W20250111 02:57:22.084527 19590 log.cc:926] Time spent T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
I20250111 02:57:22.089323 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28958a1018344a48d18e6c0f03cd0a3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:22.090180 18910 raft_consensus.cc:3054] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:22.095419 19562 leader_election.cc:290] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:22.101598 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbb63ab78f84995ae6d9fb2aae78dcf" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:22.102456 18909 raft_consensus.cc:3054] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:22.111222 19564 leader_election.cc:290] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:22.113957 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29561e984e6d40fea032075f0531dd1f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:22.114416 19330 raft_consensus.cc:357] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.115401 19330 raft_consensus.cc:383] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:22.115732 19330 raft_consensus.cc:738] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:22.117008 18625 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]). Expected version 2 got 0
I20250111 02:57:22.117439 19588 raft_consensus.cc:491] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:22.096247 19195 raft_consensus.cc:2463] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:22.117893 19588 raft_consensus.cc:513] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.120496 19588 leader_election.cc:290] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:22.122277 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8128b3941d54ce2a72bf2ae762a774f" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:22.123106 18908 raft_consensus.cc:2388] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0cedab82b154d62b78809edc6f28bfa in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:22.124605 18909 raft_consensus.cc:2463] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:22.126269 18989 leader_election.cc:304] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:22.129773 18910 raft_consensus.cc:2463] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:22.131448 19564 raft_consensus.cc:2798] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:22.132256 19564 raft_consensus.cc:695] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:22.133363 19564 consensus_queue.cc:237] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.112553 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1802e7349934fbb9d5f7eda37c3df20" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:22.135988 19196 raft_consensus.cc:2463] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:22.118137 18625 catalog_manager.cc:5526] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: UNKNOWN } } }
I20250111 02:57:22.158208 18625 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet f1802e7349934fbb9d5f7eda37c3df20 (table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0]). Expected version 4 got 0
I20250111 02:57:22.155478 18989 leader_election.cc:304] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:22.159629 19562 raft_consensus.cc:2798] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:22.160297 19562 raft_consensus.cc:695] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:22.163337 19545 raft_consensus.cc:491] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:22.164391 19545 raft_consensus.cc:513] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.167759 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1802e7349934fbb9d5f7eda37c3df20" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:22.170082 19131 leader_election.cc:304] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:22.170516 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8128b3941d54ce2a72bf2ae762a774f" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:22.173794 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef70b417568049f0b496f4d8e5f7e0e4" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:22.174832 19054 raft_consensus.cc:2463] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:22.175205 19585 raft_consensus.cc:3054] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:22.176658 18842 leader_election.cc:304] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:22.178023 19602 raft_consensus.cc:2798] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:22.178565 19602 raft_consensus.cc:491] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:22.179103 19602 raft_consensus.cc:3054] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:22.182345 19585 raft_consensus.cc:2743] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:22.188134 19131 leader_election.cc:304] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:22.190680 19545 leader_election.cc:290] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:22.162555 19562 consensus_queue.cc:237] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.156529 19330 consensus_queue.cc:260] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.203954 19588 raft_consensus.cc:3054] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:22.168534 18628 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_0928c84124ee478d802678dfddd36317" } new_table_name: "default.Table_c9582d98645645f3aa80a65f7652cba8" modify_external_catalogs: true
I20250111 02:57:22.188849 19585 consensus_queue.cc:1035] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:22.189370 19325 ts_tablet_manager.cc:1918] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:22.211320 19325 log.cc:1198] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/3c02166674a140f29e9f655d09532ced
I20250111 02:57:22.212015 19325 ts_tablet_manager.cc:1939] T 3c02166674a140f29e9f655d09532ced P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:22.190920 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef70b417568049f0b496f4d8e5f7e0e4" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:22.192054 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1802e7349934fbb9d5f7eda37c3df20" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
W20250111 02:57:22.215426 18841 leader_election.cc:343] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:22.193202 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29561e984e6d40fea032075f0531dd1f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:22.195258 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28958a1018344a48d18e6c0f03cd0a3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:22.197134 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbb63ab78f84995ae6d9fb2aae78dcf" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:22.197768 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbb63ab78f84995ae6d9fb2aae78dcf" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:22.220906 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 114) took 1723 ms. Trace:
I20250111 02:57:22.221226 19325 rpcz_store.cc:276] 0111 02:57:20.497819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:20.497961 (+ 142us) service_pool.cc:225] Handling call
0111 02:57:21.535052 (+1037091us) ts_tablet_manager.cc:1145] Deleting tablet 3c02166674a140f29e9f655d09532ced
0111 02:57:21.554386 (+ 19334us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:22.082697 (+528311us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.189153 (+106456us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.220849 (+ 31696us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":971120}
I20250111 02:57:22.222352 19325 tablet_replica.cc:331] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:22.223400 19330 ts_tablet_manager.cc:1428] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.148s user 0.007s sys 0.001s
I20250111 02:57:22.226226 19195 raft_consensus.cc:2463] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:22.227044 19325 raft_consensus.cc:2238] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:22.227551 19325 raft_consensus.cc:2267] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:22.227847 19330 tablet_bootstrap.cc:492] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:22.229470 19325 ts_tablet_manager.cc:1905] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:22.234017 18870 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:51078 (request call id 45) took 1035 ms. Trace:
I20250111 02:57:22.240280 18629 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 1b2377893bfc426fa00d5ddacd9c5961
W20250111 02:57:22.241101 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 53064.9us
I20250111 02:57:22.242894 19029 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2377893bfc426fa00d5ddacd9c5961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:21 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:22.244674 18884 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2377893bfc426fa00d5ddacd9c5961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:21 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:22.248690 19326 ts_tablet_manager.cc:1918] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:22.249084 19326 log.cc:1198] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/b715e32011114e6286c0b8e5406258c4
I20250111 02:57:22.249766 19326 ts_tablet_manager.cc:1939] T b715e32011114e6286c0b8e5406258c4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:22.257010 19605 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:22.257694 19605 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:22.262660 19588 raft_consensus.cc:2743] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:22.264678 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28958a1018344a48d18e6c0f03cd0a3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:22.265271 19167 tablet_service.cc:1467] Processing CreateTablet for tablet 934c900baf1b4000af7d7ec60cbb3591 (DEFAULT_TABLE table=default.Table_4f96c1244592444a994b5c5631eaaadc [id=4c817a432cf649799a2589f12a320ebd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:22.266304 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 934c900baf1b4000af7d7ec60cbb3591. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:22.266834 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ef3f45e5784a29b3b32cc0de6bac84" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:22.270763 19322 tablet_bootstrap.cc:492] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:22.270867 19169 tablet_service.cc:1467] Processing CreateTablet for tablet d837bd001aa649e9b3ca94f7cdd9f0fe (DEFAULT_TABLE table=default.Table_b6421db891c148c688d25610559fe16f [id=190cfd157c8d4bd19a8f40cfd80925c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:22.271396 19322 ts_tablet_manager.cc:1397] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.289s user 0.012s sys 0.020s
I20250111 02:57:22.271996 19176 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b302fcba27244c193586bc25fc9305f with delete_type TABLET_DATA_DELETED (replaced by tablet 13f0a0e7698e49a5a57562dc8ad52cd5) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:22.274037 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe4e60db4244a9a9696153b630c575b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:22.274847 19165 tablet_service.cc:1514] Processing DeleteTablet for tablet b54311f4269d4575b8998d7f545000e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 845da3ee0863430ea3fe9de8c84f2c17) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:22.275482 19171 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e06e867266948df8eb35a85c58f578f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:22.275986 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet d8128b3941d54ce2a72bf2ae762a774f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:22.279495 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d837bd001aa649e9b3ca94f7cdd9f0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:22.234519 18870 rpcz_store.cc:276] 0111 02:57:21.198281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.198409 (+ 128us) service_pool.cc:225] Handling call
0111 02:57:21.578614 (+380205us) spinlock_profiling.cc:243] Waited 65.3 ms on lock 0x7b54001901a8. stack: 00007f02faaf7853 00007f02faaf7750 00007f02fa66c584 00000000004fcd24 00000000004fccb9 00007f0301e0ea81 00007f0301e9a151 00007f0302a62690 00007f0301300e14 00007f0301300da2 00007f0301300cd1 00007f0301300c4c 00007f03012fff02 00007f02ff83fdfc 00007f02ff83f236 00007f02ff83ebdf
0111 02:57:22.233918 (+655304us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":150246272}
W20250111 02:57:22.296478 18627 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4273afc264843029ed0baf1fc5c8a3e consensus state has no leader
I20250111 02:57:22.297330 18629 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_26836b21f0f84973b442a4966f97d1ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:22.300038 18629 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26836b21f0f84973b442a4966f97d1ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:22.311113 18627 master_service.cc:946] ReplaceTablet: received request to replace tablet 3052e9707de14e1f9e38236d56908ed3 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:22.291396 19321 tablet_bootstrap.cc:492] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:22.311839 19321 ts_tablet_manager.cc:1397] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.385s user 0.008s sys 0.015s
I20250111 02:57:22.288530 19602 raft_consensus.cc:513] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.321630 19321 raft_consensus.cc:357] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.322315 19321 raft_consensus.cc:383] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:22.322571 19321 raft_consensus.cc:738] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:22.329552 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 193 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:22.332868 19605 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:22.335467 19177 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2377893bfc426fa00d5ddacd9c5961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:21 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:22.337265 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef70b417568049f0b496f4d8e5f7e0e4" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
W20250111 02:57:22.339293 18841 leader_election.cc:343] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:22.344121 18882 consensus_meta.cc:287] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c: Time spent flushing consensus metadata: real 0.703s user 0.009s sys 0.000s
W20250111 02:57:22.352000 18695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111)
I20250111 02:57:22.352815 19321 consensus_queue.cc:260] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.356503 19321 ts_tablet_manager.cc:1428] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20250111 02:57:22.361238 18621 master_service.cc:946] ReplaceTablet: received request to replace tablet 2cd364f9a6894d19826deb59a8d8a612 from {username='slave'} at 127.0.0.1:47522
W20250111 02:57:22.368961 18986 leader_election.cc:343] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:22.373448 18986 leader_election.cc:343] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:22.374694 19330 tablet_bootstrap.cc:654] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:22.347792 19322 raft_consensus.cc:357] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.375337 19322 raft_consensus.cc:383] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:22.375674 19322 raft_consensus.cc:738] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:22.376570 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 3c02166674a140f29e9f655d09532ced (table default.Table_c2796c2dda744b6c8c8b1b8a7bed2a19 [id=820cc3b845124b1ea333e55aa6537fad]) successfully deleted
I20250111 02:57:22.376500 19322 consensus_queue.cc:260] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.377703 19602 leader_election.cc:290] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:22.386291 19327 ts_tablet_manager.cc:1918] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:22.390677 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef70b417568049f0b496f4d8e5f7e0e4" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:22.391402 19052 raft_consensus.cc:3054] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:22.393330 18986 leader_election.cc:343] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:22.394840 18986 leader_election.cc:343] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:22.395793 18986 leader_election.cc:343] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:22.398609 18986 leader_election.cc:343] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:22.401729 18986 leader_election.cc:343] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:22.402402 18986 leader_election.cc:304] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f; no voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:22.403576 19533 raft_consensus.cc:2743] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:22.408044 19588 raft_consensus.cc:491] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:22.408824 19588 raft_consensus.cc:513] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:22.410650 18695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.126:45115, user_credentials={real_user=slave}} blocked reactor thread for 58962.7us
W20250111 02:57:22.414296 19133 leader_election.cc:343] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 616655aa75824fffa46f81c4f95d5159
I20250111 02:57:22.415784 19533 consensus_queue.cc:1035] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:22.414996 19588 leader_election.cc:290] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:22.419708 19444 raft_consensus.cc:491] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:22.420187 19444 raft_consensus.cc:513] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.422395 19444 leader_election.cc:290] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:22.422677 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 98) took 3030 ms. Trace:
W20250111 02:57:22.431563 18695 leader_election.cc:336] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115): Network error: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111)
I20250111 02:57:22.442997 18622 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_799d0b8328de40a7bacb58ea433fec72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:22.444904 18622 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_799d0b8328de40a7bacb58ea433fec72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:22.450579 18623 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_8292a1a0b58247ca951f8f8a951a83ad" } modify_external_catalogs: true
I20250111 02:57:22.451294 18623 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_8292a1a0b58247ca951f8f8a951a83ad" } modify_external_catalogs: true
I20250111 02:57:22.452380 19192 raft_consensus.cc:3054] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:22.425995 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe4e60db4244a9a9696153b630c575b" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:22.473315 18909 raft_consensus.cc:2388] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0cedab82b154d62b78809edc6f28bfa in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:22.430918 19322 ts_tablet_manager.cc:1428] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.159s user 0.005s sys 0.002s
I20250111 02:57:22.432999 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616655aa75824fffa46f81c4f95d5159" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:22.475499 19322 tablet_bootstrap.cc:492] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:22.475337 19054 raft_consensus.cc:2463] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:22.437897 19481 consensus_queue.cc:1035] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:22.476635 19321 tablet_bootstrap.cc:492] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:22.478003 19327 log.cc:1198] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/066c87476a854e169fd4f5f1cdbb49f7
I20250111 02:57:22.478739 19327 ts_tablet_manager.cc:1939] T 066c87476a854e169fd4f5f1cdbb49f7 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:22.485708 19325 ts_tablet_manager.cc:1918] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:22.486546 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe4e60db4244a9a9696153b630c575b" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:22.488481 19326 rpcz_store.cc:276] 0111 02:57:19.392311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:19.394045 (+ 1734us) service_pool.cc:225] Handling call
0111 02:57:21.204512 (+1810467us) ts_tablet_manager.cc:1145] Deleting tablet b715e32011114e6286c0b8e5406258c4
0111 02:57:21.204574 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:21.656106 (+451532us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.248452 (+592346us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.422604 (+174152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1809823}
I20250111 02:57:22.490331 19326 tablet_replica.cc:331] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:22.490257 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet b715e32011114e6286c0b8e5406258c4 (table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]) successfully deleted
W20250111 02:57:22.494930 18614 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b4273afc264843029ed0baf1fc5c8a3e (table default.Table_4e3a79a5d2184687a8c51fd2702c5fd0 [id=73c59c0d9864497fbc9f19e265d8283f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b4273afc264843029ed0baf1fc5c8a3e consensus state has no leader
I20250111 02:57:22.496263 19481 consensus_queue.cc:1035] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:22.496762 19326 raft_consensus.cc:2238] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:22.497747 19326 pending_rounds.cc:62] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f: Trying to abort 1 pending ops.
I20250111 02:57:22.433440 18695 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:22.507860 19605 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250111 02:57:22.508066 18695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.126:45115, user_credentials={real_user=slave}} blocked reactor thread for 76545.6us
I20250111 02:57:22.489464 19131 leader_election.cc:304] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:22.528512 19131 leader_election.cc:304] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:22.529385 19326 pending_rounds.cc:69] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112967121051713536 op_type: NO_OP noop_request { }
I20250111 02:57:22.529990 19585 raft_consensus.cc:2798] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:22.530256 18630 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 0267a92995ea460786dc26813b5895cf
I20250111 02:57:22.530416 19585 raft_consensus.cc:491] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:22.530503 19326 raft_consensus.cc:2883] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:57:22.530778 19585 raft_consensus.cc:3054] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:22.530900 19326 raft_consensus.cc:2267] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:22.499454 19481 consensus_queue.cc:1035] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:22.535085 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet 0267a92995ea460786dc26813b5895cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:22 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:22.537371 19444 raft_consensus.cc:2743] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:22.543690 19326 ts_tablet_manager.cc:1905] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:22.502205 19562 raft_consensus.cc:491] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0cedab82b154d62b78809edc6f28bfa)
I20250111 02:57:22.494895 19052 raft_consensus.cc:2463] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:22.509581 19634 tablet.cc:1721] T b5148fa2e6504561bd1b1a19e48993d7 P 54628fbcec2f4359aa9e26a734d4d29f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250111 02:57:22.546136 19562 raft_consensus.cc:513] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.548471 19562 leader_election.cc:290] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:22.550845 19325 log.cc:1198] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/f610a4ff1cc34a928f2db73f14834b25
I20250111 02:57:22.551565 19325 ts_tablet_manager.cc:1939] T f610a4ff1cc34a928f2db73f14834b25 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:22.553380 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 115) took 2054 ms. Trace:
W20250111 02:57:22.559459 18628 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2873325cece64db6a673d4fe08eb7409 has no consensus state
I20250111 02:57:22.541407 19585 raft_consensus.cc:513] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.564709 19321 tablet_bootstrap.cc:654] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:22.568809 19585 leader_election.cc:290] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:22.569275 19325 rpcz_store.cc:276] 0111 02:57:20.498788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:20.498984 (+ 196us) service_pool.cc:225] Handling call
0111 02:57:22.222205 (+1723221us) ts_tablet_manager.cc:1145] Deleting tablet f610a4ff1cc34a928f2db73f14834b25
0111 02:57:22.222246 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:22.346494 (+124248us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.485480 (+138986us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.553321 (+ 67841us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1658101}
I20250111 02:57:22.570164 18621 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]). Expected version 2 got 0
I20250111 02:57:22.570371 19325 tablet_replica.cc:331] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:22.571054 19325 raft_consensus.cc:2238] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:22.571512 19325 raft_consensus.cc:2267] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:22.573619 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet f610a4ff1cc34a928f2db73f14834b25 (table default.Table_fcfe588eae964e7b9f5af53b5728e639 [id=f1373922978a4598b888e1863525942e]) successfully deleted
I20250111 02:57:22.573724 19326 ts_tablet_manager.cc:1918] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:22.574084 19326 log.cc:1198] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/149157c1847a497e88a3136a9c1983e9
I20250111 02:57:22.574795 19326 ts_tablet_manager.cc:1939] T 149157c1847a497e88a3136a9c1983e9 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:22.576818 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 95) took 3201 ms. Trace:
I20250111 02:57:22.577121 19326 rpcz_store.cc:276] 0111 02:57:19.391238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:19.394766 (+ 3528us) service_pool.cc:225] Handling call
0111 02:57:22.490160 (+3095394us) ts_tablet_manager.cc:1145] Deleting tablet 149157c1847a497e88a3136a9c1983e9
0111 02:57:22.490209 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:22.560792 (+ 70583us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.573490 (+ 12698us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.576748 (+ 3258us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3094710}
I20250111 02:57:22.578106 19326 tablet_replica.cc:331] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:22.578871 19326 raft_consensus.cc:2238] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:22.579324 19326 pending_rounds.cc:62] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f: Trying to abort 1 pending ops.
I20250111 02:57:22.579521 19326 pending_rounds.cc:69] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112967118953123840 op_type: NO_OP noop_request { }
I20250111 02:57:22.579874 19326 raft_consensus.cc:2883] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:57:22.580181 19326 raft_consensus.cc:2267] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:22.582116 19326 ts_tablet_manager.cc:1905] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:22.582695 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 149157c1847a497e88a3136a9c1983e9 (table default.Table_d76c4a77f7d244478357d5a5fc3cdda6 [id=b46d11174b994b818737139b9b2eb13a]) successfully deleted
I20250111 02:57:22.583487 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616655aa75824fffa46f81c4f95d5159" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:22.584077 19052 raft_consensus.cc:3054] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:22.585896 18986 leader_election.cc:343] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Not found: Tablet not found: 3fa8530e894c498cbbfa73a3ba1c751e
I20250111 02:57:22.591723 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet 149157c1847a497e88a3136a9c1983e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:22.592061 19029 tablet_service.cc:1514] Processing DeleteTablet for tablet ef70b417568049f0b496f4d8e5f7e0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7ea78feee64340e39404cba0f1bf04db) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:22.592574 19052 raft_consensus.cc:2463] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:22.593333 19029 tablet_service.cc:1514] Processing DeleteTablet for tablet c28958a1018344a48d18e6c0f03cd0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:17 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:22.594154 19321 tablet_bootstrap.cc:492] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:22.599910 19325 ts_tablet_manager.cc:1905] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:22.603904 19321 ts_tablet_manager.cc:1397] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.127s user 0.012s sys 0.000s
I20250111 02:57:22.610265 19131 leader_election.cc:304] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:22.610980 19326 ts_tablet_manager.cc:1918] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:22.611449 19326 log.cc:1198] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/3fa8530e894c498cbbfa73a3ba1c751e
I20250111 02:57:22.612270 19326 ts_tablet_manager.cc:1939] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:22.613580 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 99) took 3220 ms. Trace:
I20250111 02:57:22.613950 19326 rpcz_store.cc:276] 0111 02:57:19.393206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:19.394830 (+ 1624us) service_pool.cc:225] Handling call
0111 02:57:22.577951 (+3183121us) ts_tablet_manager.cc:1145] Deleting tablet 3fa8530e894c498cbbfa73a3ba1c751e
0111 02:57:22.577993 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:22.600866 (+ 22873us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.610726 (+ 9860us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.613515 (+ 2789us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3181749}
I20250111 02:57:22.615365 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 3fa8530e894c498cbbfa73a3ba1c751e (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19]) successfully deleted
I20250111 02:57:22.615417 19321 raft_consensus.cc:357] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.617381 19321 raft_consensus.cc:383] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:22.617655 19321 raft_consensus.cc:738] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:22.614581 19585 raft_consensus.cc:2798] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:22.620447 19167 tablet_service.cc:1514] Processing DeleteTablet for tablet 29561e984e6d40fea032075f0531dd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:22.623369 19169 tablet_service.cc:1514] Processing DeleteTablet for tablet 0267a92995ea460786dc26813b5895cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:22 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:22.624236 19330 tablet_bootstrap.cc:492] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:22.624768 19330 ts_tablet_manager.cc:1397] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.398s user 0.014s sys 0.000s
I20250111 02:57:22.619134 19585 raft_consensus.cc:695] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:22.631682 19321 consensus_queue.cc:260] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.633975 19330 raft_consensus.cc:357] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.634882 19330 raft_consensus.cc:383] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:22.635567 19330 raft_consensus.cc:738] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:22.639808 19330 consensus_queue.cc:260] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.480737 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 118) took 1806 ms. Trace:
I20250111 02:57:22.644917 19330 ts_tablet_manager.cc:1428] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.019s user 0.009s sys 0.000s
I20250111 02:57:22.645139 19321 ts_tablet_manager.cc:1428] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.041s user 0.006s sys 0.001s
I20250111 02:57:22.637481 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 100) took 3242 ms. Trace:
I20250111 02:57:22.645870 19326 rpcz_store.cc:276] 0111 02:57:19.395555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:19.396378 (+ 823us) service_pool.cc:225] Handling call
0111 02:57:22.616367 (+3219989us) ts_tablet_manager.cc:1145] Deleting tablet f610a4ff1cc34a928f2db73f14834b25
0111 02:57:22.632696 (+ 16329us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:22.637408 (+ 4712us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3219312}
I20250111 02:57:22.646857 19326 tablet_replica.cc:331] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
W20250111 02:57:22.648267 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet f610a4ff1cc34a928f2db73f14834b25 because the tablet was not found. No further retry: Not found: Tablet not found: f610a4ff1cc34a928f2db73f14834b25
I20250111 02:57:22.653286 19326 raft_consensus.cc:2238] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:22.653772 19326 raft_consensus.cc:2267] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:22.653791 19330 tablet_bootstrap.cc:492] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:22.657065 18624 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_24a171ea51bb46648f255e69a57f2b36" } new_table_name: "default.Table_5bc6e85b9ba443aa8ddc113f6b5debbc" modify_external_catalogs: true
I20250111 02:57:22.659963 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_3996a48b6a884d22b0b936f19ba058a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:22.661607 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3996a48b6a884d22b0b936f19ba058a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:22.629839 19585 consensus_queue.cc:237] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.660473 19638 tablet.cc:1721] T b5148fa2e6504561bd1b1a19e48993d7 P d0cedab82b154d62b78809edc6f28bfa: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250111 02:57:22.685281 19321 tablet_bootstrap.cc:492] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:22.687536 19326 ts_tablet_manager.cc:1905] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:22.709528 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet 0267a92995ea460786dc26813b5895cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:22 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:22.710211 18842 leader_election.cc:304] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters: d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:22.711293 19588 raft_consensus.cc:491] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:22.711812 19588 raft_consensus.cc:513] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.712801 19326 ts_tablet_manager.cc:1918] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:22.713135 19326 log.cc:1198] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/92aa64b2de5746d1b12cc93a10c9d56e
I20250111 02:57:22.714874 19588 leader_election.cc:290] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:22.715162 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29561e984e6d40fea032075f0531dd1f" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:22.713768 19326 ts_tablet_manager.cc:1939] T 92aa64b2de5746d1b12cc93a10c9d56e P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:22.719112 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 101) took 3321 ms. Trace:
I20250111 02:57:22.719430 19326 rpcz_store.cc:276] 0111 02:57:19.398027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:19.407668 (+ 9641us) service_pool.cc:225] Handling call
0111 02:57:22.646706 (+3239038us) ts_tablet_manager.cc:1145] Deleting tablet 92aa64b2de5746d1b12cc93a10c9d56e
0111 02:57:22.646748 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:22.704214 (+ 57466us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.712582 (+ 8368us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.719016 (+ 6434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3238257}
I20250111 02:57:22.720820 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 92aa64b2de5746d1b12cc93a10c9d56e (table default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3]) successfully deleted
I20250111 02:57:22.720997 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 111) took 2992 ms. Trace:
I20250111 02:57:22.721305 19326 rpcz_store.cc:276] 0111 02:57:19.728196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:19.885127 (+156931us) service_pool.cc:225] Handling call
0111 02:57:22.720640 (+2835513us) ts_tablet_manager.cc:1145] Deleting tablet 32afc2f158254811836fcf00f7f409ed
0111 02:57:22.720681 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:22.720943 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2834823}
W20250111 02:57:22.722460 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 32afc2f158254811836fcf00f7f409ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32afc2f158254811836fcf00f7f409ed already in progress: creating tablet
I20250111 02:57:22.723321 19326 tablet_replica.cc:331] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:22.724066 19326 raft_consensus.cc:2238] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:22.724748 19326 pending_rounds.cc:62] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f: Trying to abort 1 pending ops.
I20250111 02:57:22.724967 19326 pending_rounds.cc:69] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112967136039559168 op_type: NO_OP noop_request { }
I20250111 02:57:22.725332 19326 raft_consensus.cc:2883] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:57:22.725659 19326 raft_consensus.cc:2267] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:22.727919 19326 ts_tablet_manager.cc:1905] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:22.733546 19133 leader_election.cc:343] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 616655aa75824fffa46f81c4f95d5159
I20250111 02:57:22.737681 18626 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_75e5f05a97b14bf883909660868761fd" } modify_external_catalogs: true
I20250111 02:57:22.738238 19325 ts_tablet_manager.cc:1918] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:22.738260 18626 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_75e5f05a97b14bf883909660868761fd" } modify_external_catalogs: true
I20250111 02:57:22.738600 19325 log.cc:1198] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/ede92343d9b54c29b8173a5eafcf68c6
I20250111 02:57:22.644876 19327 rpcz_store.cc:276] 0111 02:57:20.674666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:20.674771 (+ 105us) service_pool.cc:225] Handling call
0111 02:57:21.365678 (+690907us) ts_tablet_manager.cc:1145] Deleting tablet 066c87476a854e169fd4f5f1cdbb49f7
0111 02:57:21.365713 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:22.096830 (+731117us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.386071 (+289241us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.480678 (+ 94607us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":82160,"tablet-delete.queue_time_us":690300}
I20250111 02:57:22.750650 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_02a1e7d9f9094de5b4a61b79970284a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:22.751536 19321 tablet_bootstrap.cc:654] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:22.752198 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02a1e7d9f9094de5b4a61b79970284a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:22.754597 19325 ts_tablet_manager.cc:1939] T ede92343d9b54c29b8173a5eafcf68c6 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:22.763868 19481 consensus_queue.cc:1035] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:22.768144 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 118) took 2154 ms. Trace:
I20250111 02:57:22.768496 19325 rpcz_store.cc:276] 0111 02:57:20.613534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:20.613666 (+ 132us) service_pool.cc:225] Handling call
0111 02:57:22.570222 (+1956556us) ts_tablet_manager.cc:1145] Deleting tablet ede92343d9b54c29b8173a5eafcf68c6
0111 02:57:22.570262 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:22.671487 (+101225us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.738019 (+ 66532us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.768076 (+ 30057us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1955880}
I20250111 02:57:22.773363 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet ede92343d9b54c29b8173a5eafcf68c6 (table default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]) successfully deleted
I20250111 02:57:22.774978 19325 tablet_replica.cc:331] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:22.775724 19325 raft_consensus.cc:2238] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:22.776620 19325 raft_consensus.cc:2267] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:22.780843 18627 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 3052e9707de14e1f9e38236d56908ed3
I20250111 02:57:22.782094 18627 catalog_manager.cc:6596] ReplaceTablet: tablet 3052e9707de14e1f9e38236d56908ed3 deleted and replaced by tablet f31ad2e1895d4a26b8e9f7cd7366ef81
I20250111 02:57:22.784792 19029 tablet_service.cc:1514] Processing DeleteTablet for tablet 3052e9707de14e1f9e38236d56908ed3 with delete_type TABLET_DATA_DELETED (replaced by tablet f31ad2e1895d4a26b8e9f7cd7366ef81) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:22.805150 19325 ts_tablet_manager.cc:1905] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:22.736337 19545 raft_consensus.cc:2798] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:22.735034 19602 raft_consensus.cc:491] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:22.808377 19169 tablet_service.cc:1514] Processing DeleteTablet for tablet 3052e9707de14e1f9e38236d56908ed3 with delete_type TABLET_DATA_DELETED (replaced by tablet f31ad2e1895d4a26b8e9f7cd7366ef81) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:22.821800 19326 ts_tablet_manager.cc:1918] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:22.822181 19326 log.cc:1198] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/0dbb63ab78f84995ae6d9fb2aae78dcf
I20250111 02:57:22.822863 19326 ts_tablet_manager.cc:1939] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:22.824760 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 113) took 2489 ms. Trace:
I20250111 02:57:22.825068 19326 rpcz_store.cc:276] 0111 02:57:20.335640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:20.335793 (+ 153us) service_pool.cc:225] Handling call
0111 02:57:22.723171 (+2387378us) ts_tablet_manager.cc:1145] Deleting tablet 0dbb63ab78f84995ae6d9fb2aae78dcf
0111 02:57:22.723211 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:22.744002 (+ 20791us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.821542 (+ 77540us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.824691 (+ 3149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2386373}
I20250111 02:57:22.826257 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 0dbb63ab78f84995ae6d9fb2aae78dcf (table default.Table_154c30276c8d4a8e805dc2fb85867bbf [id=9d3552a625804f0a9350454f3b5dc2e8]) successfully deleted
I20250111 02:57:22.830394 19321 tablet_bootstrap.cc:492] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:22.830819 19321 ts_tablet_manager.cc:1397] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.146s user 0.015s sys 0.000s
I20250111 02:57:22.807974 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29561e984e6d40fea032075f0531dd1f" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:22.832569 18906 raft_consensus.cc:2388] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0cedab82b154d62b78809edc6f28bfa in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:22.833483 19321 raft_consensus.cc:357] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.834141 19321 raft_consensus.cc:383] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:22.834388 19321 raft_consensus.cc:738] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:22.808718 19602 raft_consensus.cc:513] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.837567 19602 leader_election.cc:290] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:22.838207 19602 raft_consensus.cc:491] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:22.843933 19133 leader_election.cc:304] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:22.845564 19585 raft_consensus.cc:3054] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:22.851305 18989 leader_election.cc:343] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 3fa8530e894c498cbbfa73a3ba1c751e
I20250111 02:57:22.851922 18989 leader_election.cc:304] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f; no voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:22.835004 19321 consensus_queue.cc:260] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:22.852707 18989 raft_consensus.cc:2681] T 3fa8530e894c498cbbfa73a3ba1c751e P 54628fbcec2f4359aa9e26a734d4d29f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250111 02:57:22.856523 19321 ts_tablet_manager.cc:1428] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20250111 02:57:22.857580 19321 tablet_bootstrap.cc:492] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:22.866525 19588 raft_consensus.cc:491] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:22.867192 19588 raft_consensus.cc:513] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.869820 19588 leader_election.cc:290] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:22.872362 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef70b417568049f0b496f4d8e5f7e0e4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:22.873431 19052 raft_consensus.cc:2388] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0cedab82b154d62b78809edc6f28bfa in current term 1: Already voted for candidate 8221604733fe4fc0a1fc32de8d6af71c in this term.
I20250111 02:57:22.875490 19326 tablet_replica.cc:331] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:22.876228 19326 raft_consensus.cc:2238] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:22.876681 19326 raft_consensus.cc:2267] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:22.877106 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef70b417568049f0b496f4d8e5f7e0e4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:22.878409 19326 ts_tablet_manager.cc:1905] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:22.881557 19321 tablet_bootstrap.cc:654] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:22.838719 19602 raft_consensus.cc:513] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.892112 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e8c72c99ac4c369957b571d74baffe" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:22.893043 19052 raft_consensus.cc:2463] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:22.898980 19562 consensus_queue.cc:1035] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:22.906245 18842 leader_election.cc:304] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:22.913336 19325 ts_tablet_manager.cc:1918] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:22.913857 19325 log.cc:1198] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/066c87476a854e169fd4f5f1cdbb49f7
I20250111 02:57:22.914765 19325 ts_tablet_manager.cc:1939] T 066c87476a854e169fd4f5f1cdbb49f7 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:22.917143 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 119) took 2303 ms. Trace:
I20250111 02:57:22.917486 19325 rpcz_store.cc:276] 0111 02:57:20.613868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:20.614506 (+ 638us) service_pool.cc:225] Handling call
0111 02:57:22.774800 (+2160294us) ts_tablet_manager.cc:1145] Deleting tablet 066c87476a854e169fd4f5f1cdbb49f7
0111 02:57:22.774847 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:22.860384 (+ 85537us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.902608 (+ 42224us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.917075 (+ 14467us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2159706}
I20250111 02:57:22.918848 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 066c87476a854e169fd4f5f1cdbb49f7 (table default.Table_211b4c873fa34d6b8d9f4c1a303d5b28 [id=6c02a835750c4952a48c4089e4fe120c]) successfully deleted
I20250111 02:57:22.920852 19533 raft_consensus.cc:491] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:22.921468 19533 raft_consensus.cc:513] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:22.931546 18616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.126:45115: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20250111 02:57:22.924755 19533 leader_election.cc:290] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:22.931782 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40fdec982cf643c68ef90798f4f65ad4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:22.933410 18986 leader_election.cc:343] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:22.937319 19321 tablet_bootstrap.cc:492] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:22.937772 19321 ts_tablet_manager.cc:1397] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.080s user 0.009s sys 0.000s
I20250111 02:57:22.938045 19602 leader_election.cc:290] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:22.940577 19321 raft_consensus.cc:357] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.941354 19321 raft_consensus.cc:383] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:22.941628 19321 raft_consensus.cc:738] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:22.944062 19325 tablet_replica.cc:331] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:22.944880 19325 raft_consensus.cc:2238] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:22.945339 19325 raft_consensus.cc:2267] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:22.950043 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 066c87476a854e169fd4f5f1cdbb49f7 (table default.Table_211b4c873fa34d6b8d9f4c1a303d5b28 [id=6c02a835750c4952a48c4089e4fe120c]) successfully deleted
I20250111 02:57:22.952013 19321 consensus_queue.cc:260] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:22.957584 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e8c72c99ac4c369957b571d74baffe" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:22.958498 19197 raft_consensus.cc:2463] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:22.964370 19321 ts_tablet_manager.cc:1428] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20250111 02:57:22.965009 19325 ts_tablet_manager.cc:1905] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:22.983614 19321 tablet_bootstrap.cc:492] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:22.992322 19326 ts_tablet_manager.cc:1918] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:22.992718 19326 log.cc:1198] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/3c02166674a140f29e9f655d09532ced
I20250111 02:57:22.993419 19326 ts_tablet_manager.cc:1939] T 3c02166674a140f29e9f655d09532ced P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:23.006522 19322 tablet_bootstrap.cc:700] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent opening tablet: real 0.211s user 0.001s sys 0.000s
I20250111 02:57:23.007015 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 115) took 2586 ms. Trace:
I20250111 02:57:23.007725 19322 tablet_bootstrap.cc:654] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:23.007426 19326 rpcz_store.cc:276] 0111 02:57:20.420758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:20.423579 (+ 2821us) service_pool.cc:225] Handling call
0111 02:57:22.875328 (+2451749us) ts_tablet_manager.cc:1145] Deleting tablet 3c02166674a140f29e9f655d09532ced
0111 02:57:22.875373 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:22.945649 (+ 70276us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:22.992095 (+ 46446us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:23.006930 (+ 14835us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2443373}
I20250111 02:57:23.013777 19585 raft_consensus.cc:2743] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:23.016010 19321 tablet_bootstrap.cc:654] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:23.015811 19602 raft_consensus.cc:2798] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 02:57:23.016634 19602 raft_consensus.cc:491] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:23.017251 19602 raft_consensus.cc:3054] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:57:23.017442 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 118) took 2504 ms. Trace:
I20250111 02:57:23.017779 19326 rpcz_store.cc:276] 0111 02:57:20.513340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:20.515164 (+ 1824us) service_pool.cc:225] Handling call
0111 02:57:23.017159 (+2501995us) ts_tablet_manager.cc:1145] Deleting tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7
0111 02:57:23.017202 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:23.017381 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2501394}
I20250111 02:57:23.019295 19326 tablet_replica.cc:331] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:23.021836 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet 3052e9707de14e1f9e38236d56908ed3 with delete_type TABLET_DATA_DELETED (replaced by tablet f31ad2e1895d4a26b8e9f7cd7366ef81) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:23.022708 19326 raft_consensus.cc:2238] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:23.023241 19326 raft_consensus.cc:2267] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:23.038205 19325 ts_tablet_manager.cc:1918] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:23.038697 19325 log.cc:1198] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/5118d4b9cfbb49e8af6af56180858336
I20250111 02:57:23.039319 19327 tablet_replica.cc:331] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:23.039628 19325 ts_tablet_manager.cc:1939] T 5118d4b9cfbb49e8af6af56180858336 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:23.040936 19327 raft_consensus.cc:2238] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:23.041575 19327 raft_consensus.cc:2267] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:23.042563 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ef3f45e5784a29b3b32cc0de6bac84" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:23.046636 19326 ts_tablet_manager.cc:1905] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:23.048013 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 120) took 2433 ms. Trace:
I20250111 02:57:23.054466 19650 tablet.cc:1721] T b5148fa2e6504561bd1b1a19e48993d7 P 8221604733fe4fc0a1fc32de8d6af71c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250111 02:57:23.061738 18986 consensus_peers.cc:487] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5118d4b9cfbb49e8af6af56180858336. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:23.063912 18622 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_5f09ae109ece4cd780bf64aa3d7b35c4" } new_table_name: "default.Table_b2b20f00f5994021a490df79bca9399a" modify_external_catalogs: true
I20250111 02:57:23.068018 19325 rpcz_store.cc:276] 0111 02:57:20.614226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:20.615220 (+ 994us) service_pool.cc:225] Handling call
0111 02:57:22.943891 (+2328671us) ts_tablet_manager.cc:1145] Deleting tablet 5118d4b9cfbb49e8af6af56180858336
0111 02:57:22.943938 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:23.010432 (+ 66494us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:23.037973 (+ 27541us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:23.047925 (+ 9952us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2328119}
I20250111 02:57:23.069473 19330 tablet_bootstrap.cc:654] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:23.070502 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 5118d4b9cfbb49e8af6af56180858336 (table default.Table_39701f385f884aac9233a842fcaf93be [id=bb83163ef256406da8650e72eadf7f6c]) successfully deleted
I20250111 02:57:23.073366 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 123) took 2390 ms. Trace:
I20250111 02:57:23.076913 19325 rpcz_store.cc:276] 0111 02:57:20.682762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:20.688115 (+ 5353us) service_pool.cc:225] Handling call
0111 02:57:23.072930 (+2384815us) ts_tablet_manager.cc:1145] Deleting tablet 5cf1ce65ccf54358b95665b9764fbdb0
0111 02:57:23.072994 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:23.073305 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2360225}
I20250111 02:57:23.078270 18623 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 13f0a0e7698e49a5a57562dc8ad52cd5
W20250111 02:57:23.079782 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 5cf1ce65ccf54358b95665b9764fbdb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cf1ce65ccf54358b95665b9764fbdb0 already in progress: creating tablet
I20250111 02:57:23.082799 19169 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dbb63ab78f84995ae6d9fb2aae78dcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:32868
W20250111 02:57:23.084290 18624 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74592f80afab4513a9eb2eac351c4de5 consensus state has no leader
I20250111 02:57:23.092324 19325 tablet_replica.cc:331] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:23.094475 19325 raft_consensus.cc:2238] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:57:23.103637 19325 raft_consensus.cc:2267] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:23.107159 19321 tablet_bootstrap.cc:492] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:23.107574 19321 ts_tablet_manager.cc:1397] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.124s user 0.012s sys 0.013s
I20250111 02:57:23.107929 19325 ts_tablet_manager.cc:1905] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:23.119233 18615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.124:34593, user_credentials={real_user=slave}} blocked reactor thread for 273228us
I20250111 02:57:23.128228 19533 consensus_queue.cc:1035] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:23.138239 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 3c02166674a140f29e9f655d09532ced (table default.Table_c2796c2dda744b6c8c8b1b8a7bed2a19 [id=820cc3b845124b1ea333e55aa6537fad]) successfully deleted
W20250111 02:57:23.138984 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 4feb96efdfc44fe4a7cd46dc8deb4fe7 because the tablet was not found. No further retry: Not found: Tablet not found: 4feb96efdfc44fe4a7cd46dc8deb4fe7
I20250111 02:57:23.110548 19321 raft_consensus.cc:357] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.139997 19321 raft_consensus.cc:383] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:23.140290 19321 raft_consensus.cc:738] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:23.140910 19321 consensus_queue.cc:260] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.142289 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_d234d85b2b234eef9e672be1cb4a1923"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:23.143844 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d234d85b2b234eef9e672be1cb4a1923 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:23.154665 18621 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_37a329a4875c463ea093ad5ab23e9cf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:23.156670 18621 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37a329a4875c463ea093ad5ab23e9cf6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:23.160709 19325 ts_tablet_manager.cc:1918] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250111 02:57:23.161110 19325 log.cc:1198] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/5baef8688ede42fca28cfb6b0bdeefa6
I20250111 02:57:23.168763 19321 ts_tablet_manager.cc:1428] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.061s user 0.006s sys 0.000s
I20250111 02:57:23.168429 18630 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet f1802e7349934fbb9d5f7eda37c3df20 (table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0]). Expected version 4 got 0
I20250111 02:57:23.174834 18626 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet b26e5db463e94d919640a07a2f8574a8
I20250111 02:57:23.187044 19029 tablet_service.cc:1514] Processing DeleteTablet for tablet b26e5db463e94d919640a07a2f8574a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:22 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:23.169119 19325 ts_tablet_manager.cc:1939] T 5baef8688ede42fca28cfb6b0bdeefa6 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:23.191591 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 128) took 2053 ms. Trace:
I20250111 02:57:23.183050 19169 tablet_service.cc:1514] Processing DeleteTablet for tablet b26e5db463e94d919640a07a2f8574a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:22 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:23.191936 19325 rpcz_store.cc:276] 0111 02:57:21.138496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.186693 (+ 48197us) service_pool.cc:225] Handling call
0111 02:57:23.080585 (+1893892us) ts_tablet_manager.cc:1145] Deleting tablet 5baef8688ede42fca28cfb6b0bdeefa6
0111 02:57:23.087059 (+ 6474us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:23.139523 (+ 52464us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:23.160486 (+ 20963us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:23.191522 (+ 31036us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1882626}
I20250111 02:57:23.193302 19325 tablet_replica.cc:331] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:23.194038 19325 raft_consensus.cc:2238] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:23.194689 19325 pending_rounds.cc:62] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa: Trying to abort 1 pending ops.
I20250111 02:57:23.194897 19325 pending_rounds.cc:69] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112967138029367296 op_type: NO_OP noop_request { }
I20250111 02:57:23.195317 19325 raft_consensus.cc:2883] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:57:23.195628 19325 raft_consensus.cc:2267] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:23.197886 19325 ts_tablet_manager.cc:1905] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:23.184369 19654 log.cc:926] Time spent T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa: Append to log took a long time: real 0.126s user 0.001s sys 0.000s
I20250111 02:57:23.203301 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 5baef8688ede42fca28cfb6b0bdeefa6 (table default.Table_7f4578da9f5e418c9d67a535471b45cd [id=154c37dc0dba44bdab5e0315756dc7cf]) successfully deleted
W20250111 02:57:23.213052 18622 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc7158a7fc624fa4bc816ac3721c610d consensus state has no leader
I20250111 02:57:23.219259 19321 tablet_bootstrap.cc:492] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:23.234515 19192 raft_consensus.cc:3054] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:23.241135 19325 ts_tablet_manager.cc:1918] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:23.241503 19325 log.cc:1198] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/616655aa75824fffa46f81c4f95d5159
I20250111 02:57:23.241855 19321 tablet_bootstrap.cc:654] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:23.242210 19325 ts_tablet_manager.cc:1939] T 616655aa75824fffa46f81c4f95d5159 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:23.243806 19672 tablet.cc:1721] T f1802e7349934fbb9d5f7eda37c3df20 P d0cedab82b154d62b78809edc6f28bfa: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250111 02:57:23.249886 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 131) took 1914 ms. Trace:
I20250111 02:57:23.249635 19533 consensus_queue.cc:1035] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:23.250247 19325 rpcz_store.cc:276] 0111 02:57:21.335475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.335617 (+ 142us) service_pool.cc:225] Handling call
0111 02:57:23.193154 (+1857537us) ts_tablet_manager.cc:1145] Deleting tablet 616655aa75824fffa46f81c4f95d5159
0111 02:57:23.193195 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:23.223766 (+ 30571us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:23.240923 (+ 17157us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:23.249822 (+ 8899us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1856725}
I20250111 02:57:23.252125 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 616655aa75824fffa46f81c4f95d5159 (table default.Table_a58de727de864b9a8ecab09048b72fb7 [id=581224f40f0c4f03a9224f06d660c883]) successfully deleted
I20250111 02:57:23.252656 19325 tablet_replica.cc:331] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:23.253408 19325 raft_consensus.cc:2238] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:23.253866 19325 raft_consensus.cc:2267] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:23.256894 19325 ts_tablet_manager.cc:1905] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:23.292666 19321 tablet_bootstrap.cc:492] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:23.296821 19321 ts_tablet_manager.cc:1397] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.092s user 0.004s sys 0.010s
I20250111 02:57:23.300383 19321 raft_consensus.cc:357] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.301255 19325 ts_tablet_manager.cc:1918] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:23.319312 19325 log.cc:1198] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/9fe4e60db4244a9a9696153b630c575b
I20250111 02:57:23.320180 19325 ts_tablet_manager.cc:1939] T 9fe4e60db4244a9a9696153b630c575b P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:23.322150 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 144) took 1055 ms. Trace:
I20250111 02:57:23.322660 19325 rpcz_store.cc:276] 0111 02:57:22.266275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.269359 (+ 3084us) service_pool.cc:225] Handling call
0111 02:57:23.252513 (+983154us) ts_tablet_manager.cc:1145] Deleting tablet 9fe4e60db4244a9a9696153b630c575b
0111 02:57:23.252547 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:23.285679 (+ 33132us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:23.301025 (+ 15346us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:23.322091 (+ 21066us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":978044}
I20250111 02:57:23.323755 19325 tablet_replica.cc:331] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:23.324414 19325 raft_consensus.cc:2238] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:23.324388 18624 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_91137e45d2b549c6a6c6a478c43c1033" } modify_external_catalogs: true
I20250111 02:57:23.324932 18624 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_91137e45d2b549c6a6c6a478c43c1033" } modify_external_catalogs: true
I20250111 02:57:23.325387 18625 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0] alter complete (version 1)
I20250111 02:57:23.325911 18622 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_3996a48b6a884d22b0b936f19ba058a3" } new_table_name: "default.Table_4b9e85e01e1843ce831205177cbb2e0f" modify_external_catalogs: true
I20250111 02:57:23.324822 19325 raft_consensus.cc:2267] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:23.326733 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 9fe4e60db4244a9a9696153b630c575b (table default.Table_064d4893c74f4870bc45572dd481c63c [id=b0a0ff2f4d554503a18d05e0ff51402d]) successfully deleted
I20250111 02:57:23.328064 19325 ts_tablet_manager.cc:1905] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:23.329066 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_bdacd508498241408a549fdbc6704512"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:23.330580 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdacd508498241408a549fdbc6704512 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:23.333464 19330 tablet_bootstrap.cc:492] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:23.333982 19330 ts_tablet_manager.cc:1397] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.688s user 0.011s sys 0.004s
I20250111 02:57:23.319301 19321 raft_consensus.cc:383] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:23.337621 19321 raft_consensus.cc:738] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:23.338382 19321 consensus_queue.cc:260] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.336812 19330 raft_consensus.cc:357] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.342499 19330 raft_consensus.cc:383] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:23.343619 19330 raft_consensus.cc:738] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:23.343303 19321 ts_tablet_manager.cc:1428] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20250111 02:57:23.345033 19330 consensus_queue.cc:260] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.349180 19321 tablet_bootstrap.cc:492] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:23.354285 19330 ts_tablet_manager.cc:1428] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250111 02:57:23.355424 19330 tablet_bootstrap.cc:492] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:23.361685 19325 ts_tablet_manager.cc:1918] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:23.369823 19325 log.cc:1198] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/b54311f4269d4575b8998d7f545000e7
I20250111 02:57:23.370728 19325 ts_tablet_manager.cc:1939] T b54311f4269d4575b8998d7f545000e7 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:23.372766 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 145) took 1104 ms. Trace:
I20250111 02:57:23.373154 19325 rpcz_store.cc:276] 0111 02:57:22.268107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.274609 (+ 6502us) service_pool.cc:225] Handling call
0111 02:57:23.323594 (+1048985us) ts_tablet_manager.cc:1145] Deleting tablet b54311f4269d4575b8998d7f545000e7
0111 02:57:23.323644 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:23.344193 (+ 20549us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:23.361472 (+ 17279us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:23.372706 (+ 11234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1048291}
I20250111 02:57:23.374464 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet b54311f4269d4575b8998d7f545000e7 (table default.Table_292c103f235f421db8dacf834cca2c53 [id=b091f6cd63104693abbdd739bca159c3]) successfully deleted
W20250111 02:57:23.360234 18841 leader_election.cc:343] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:23.375615 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 141) took 1110 ms. Trace:
I20250111 02:57:23.375921 19325 rpcz_store.cc:276] 0111 02:57:22.265249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.273756 (+ 8507us) service_pool.cc:225] Handling call
0111 02:57:23.375283 (+1101527us) ts_tablet_manager.cc:1145] Deleting tablet 2e06e867266948df8eb35a85c58f578f
0111 02:57:23.375324 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:23.375564 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1099438}
W20250111 02:57:23.377067 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 2e06e867266948df8eb35a85c58f578f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e06e867266948df8eb35a85c58f578f already in progress: creating tablet
I20250111 02:57:23.378337 19325 tablet_replica.cc:331] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:23.379021 19325 raft_consensus.cc:2238] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:23.379483 19325 raft_consensus.cc:2267] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
W20250111 02:57:23.380499 18841 consensus_peers.cc:487] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:23.384210 19321 tablet_bootstrap.cc:654] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:23.394917 18626 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_35d7f864912b40bcb5251a90d40c7849" } modify_external_catalogs: true
I20250111 02:57:23.395510 18626 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_35d7f864912b40bcb5251a90d40c7849" } modify_external_catalogs: true
I20250111 02:57:23.395778 19321 tablet_bootstrap.cc:492] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:23.396173 19321 ts_tablet_manager.cc:1397] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.011s
I20250111 02:57:23.396590 18623 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_23a3f1c40c15458e93c2ef79a3c9e9e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:23.398161 18623 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23a3f1c40c15458e93c2ef79a3c9e9e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:23.397836 19650 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.341s user 0.001s sys 0.000s
I20250111 02:57:23.398754 19321 raft_consensus.cc:357] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.399091 19585 raft_consensus.cc:491] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:23.399629 19585 raft_consensus.cc:513] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.402235 19325 ts_tablet_manager.cc:1905] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:23.402181 19585 leader_election.cc:290] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:23.404558 19321 raft_consensus.cc:383] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:23.404971 19321 raft_consensus.cc:738] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:23.405189 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbb63ab78f84995ae6d9fb2aae78dcf" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:23.405791 19321 consensus_queue.cc:260] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.406279 18907 raft_consensus.cc:2388] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0cedab82b154d62b78809edc6f28bfa in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:23.409415 19321 ts_tablet_manager.cc:1428] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250111 02:57:23.410286 19321 tablet_bootstrap.cc:492] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:23.423653 18629 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_b2b20f00f5994021a490df79bca9399a" } new_table_name: "default.Table_019bed496592413585d882a5d46666cb" modify_external_catalogs: true
I20250111 02:57:23.429853 18627 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_2c181f66f28741b7b388a7b563b7d2f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:23.431416 18627 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c181f66f28741b7b388a7b563b7d2f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:23.434840 19325 ts_tablet_manager.cc:1918] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:23.435212 19325 log.cc:1198] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/d8128b3941d54ce2a72bf2ae762a774f
I20250111 02:57:23.438031 19325 ts_tablet_manager.cc:1939] T d8128b3941d54ce2a72bf2ae762a774f P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:23.439972 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 142) took 1174 ms. Trace:
I20250111 02:57:23.440306 19325 rpcz_store.cc:276] 0111 02:57:22.265571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.273788 (+ 8217us) service_pool.cc:225] Handling call
0111 02:57:23.377843 (+1104055us) ts_tablet_manager.cc:1145] Deleting tablet d8128b3941d54ce2a72bf2ae762a774f
0111 02:57:23.378231 (+ 388us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:23.419840 (+ 41609us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:23.434207 (+ 14367us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:23.439905 (+ 5698us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1101382}
I20250111 02:57:23.441566 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet d8128b3941d54ce2a72bf2ae762a774f (table default.Table_8b6452ba06af45bab320b50842e527d8 [id=0fa5e1f97e984916a703c7b7da0f1087]) successfully deleted
I20250111 02:57:23.442708 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 146) took 1174 ms. Trace:
I20250111 02:57:23.443084 19325 rpcz_store.cc:276] 0111 02:57:22.268561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.334787 (+ 66226us) service_pool.cc:225] Handling call
0111 02:57:23.442374 (+1107587us) ts_tablet_manager.cc:1145] Deleting tablet 1b2377893bfc426fa00d5ddacd9c5961
0111 02:57:23.442413 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:23.442657 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1106193}
I20250111 02:57:23.443624 19321 tablet_bootstrap.cc:654] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:23.444583 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 1b2377893bfc426fa00d5ddacd9c5961 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b2377893bfc426fa00d5ddacd9c5961 already in progress: creating tablet
I20250111 02:57:23.445545 19325 tablet_replica.cc:331] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:23.445456 19585 raft_consensus.cc:491] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:23.445986 19585 raft_consensus.cc:513] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.447965 19585 leader_election.cc:290] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:23.411751 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet b26e5db463e94d919640a07a2f8574a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:22 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:23.454380 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ef3f45e5784a29b3b32cc0de6bac84" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:23.448554 18842 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.000s sys 0.010s
W20250111 02:57:23.457082 18842 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.545s user 0.000s sys 0.010s
W20250111 02:57:23.457841 18842 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 551762us
W20250111 02:57:23.448441 19545 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.006s sys 0.010s
W20250111 02:57:23.459590 19545 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.628s user 0.006s sys 0.010s
I20250111 02:57:23.460044 19545 raft_consensus.cc:695] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 LEADER]: Becoming Leader. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Running, Role: LEADER
I20250111 02:57:23.461476 19545 consensus_queue.cc:237] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.456518 18907 raft_consensus.cc:2463] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:23.464170 19325 raft_consensus.cc:2238] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:23.464691 19325 raft_consensus.cc:2267] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:23.464439 19133 leader_election.cc:304] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:23.465795 19585 raft_consensus.cc:2798] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:23.466218 19585 raft_consensus.cc:491] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:23.466645 19585 raft_consensus.cc:3054] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:23.471347 19325 ts_tablet_manager.cc:1905] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:23.477159 19585 raft_consensus.cc:513] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.482388 19330 tablet_bootstrap.cc:654] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:23.483687 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ef3f45e5784a29b3b32cc0de6bac84" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:23.484380 19054 raft_consensus.cc:2463] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
W20250111 02:57:23.448287 18906 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.006s sys 0.006s
I20250111 02:57:23.486603 19671 tablet.cc:1721] T f1802e7349934fbb9d5f7eda37c3df20 P 54628fbcec2f4359aa9e26a734d4d29f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250111 02:57:23.463402 19327 ts_tablet_manager.cc:1905] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:23.435496 18910 thread.cc:635] apply [worker] (thread pool) Time spent creating pthread: real 0.915s user 0.001s sys 0.018s
W20250111 02:57:23.508246 18910 thread.cc:601] apply [worker] (thread pool) Time spent starting thread: real 0.988s user 0.001s sys 0.018s
W20250111 02:57:23.486156 18906 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.578s user 0.007s sys 0.007s
I20250111 02:57:23.487005 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ef3f45e5784a29b3b32cc0de6bac84" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:23.514822 19330 tablet_bootstrap.cc:492] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:23.515417 19330 ts_tablet_manager.cc:1397] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.160s user 0.015s sys 0.001s
I20250111 02:57:23.515249 19054 raft_consensus.cc:2463] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:23.517760 19330 raft_consensus.cc:357] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.518440 19330 raft_consensus.cc:383] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:23.518708 19330 raft_consensus.cc:738] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:23.519443 18910 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.18.20.66:58419 (request call id 68) took 1015 ms. Trace:
I20250111 02:57:23.520874 19330 consensus_queue.cc:260] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:23.526919 18614 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b4273afc264843029ed0baf1fc5c8a3e (table default.Table_4e3a79a5d2184687a8c51fd2702c5fd0 [id=73c59c0d9864497fbc9f19e265d8283f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b4273afc264843029ed0baf1fc5c8a3e consensus state has no leader
I20250111 02:57:23.533793 19330 ts_tablet_manager.cc:1428] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250111 02:57:23.537914 19330 tablet_bootstrap.cc:492] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:23.547852 19330 tablet_bootstrap.cc:654] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:23.555891 18910 rpcz_store.cc:276] 0111 02:57:22.504591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.519098 (+ 14507us) service_pool.cc:225] Handling call
0111 02:57:22.519557 (+ 459us) raft_consensus.cc:1466] Updating replica for 0 ops
0111 02:57:22.519853 (+ 296us) raft_consensus.cc:1529] Early marking committed up to index 2
0111 02:57:23.509724 (+989871us) raft_consensus.cc:1534] Triggering prepare for 0 ops
0111 02:57:23.509807 (+ 83us) raft_consensus.cc:1640] Marking committed up to 2
0111 02:57:23.509873 (+ 66us) raft_consensus.cc:1692] Filling consensus response to leader.
0111 02:57:23.519162 (+ 9289us) raft_consensus.cc:1686] UpdateReplicas() finished
0111 02:57:23.519349 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:23.593503 19029 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dbb63ab78f84995ae6d9fb2aae78dcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:36190
W20250111 02:57:23.597944 18625 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.66:37049 (request call id 35) took 948 ms (client timeout 1000 ms). Trace:
W20250111 02:57:23.601229 18625 rpcz_store.cc:269] 0111 02:57:22.650046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.650266 (+ 220us) service_pool.cc:225] Handling call
0111 02:57:22.663259 (+ 12993us) spinlock_profiling.cc:243] Waited 9.76 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d51dee 00007fd427aaffc4 00007fd427aaff52 00007fd427aafe81 00007fd427aafdfc 00007fd427aaf0b2 00007fd425d79dfc 00007fd425d79236
0111 02:57:23.324980 (+661721us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:23.325011 (+ 31us) write_op.cc:454] Released schema lock
0111 02:57:23.597879 (+272868us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:23.285819 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:23.286323 (+ 504us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:23.286361 (+ 38us) write_op.cc:435] Acquired schema lock
0111 02:57:23.286383 (+ 22us) tablet.cc:659] Decoding operations
0111 02:57:23.286740 (+ 357us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:23.286800 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:23.286835 (+ 35us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:23.287100 (+ 265us) tablet.cc:698] Row locks acquired
0111 02:57:23.287123 (+ 23us) write_op.cc:260] PREPARE: finished
0111 02:57:23.287283 (+ 160us) write_op.cc:270] Start()
0111 02:57:23.287395 (+ 112us) write_op.cc:276] Timestamp: P: 1736564243287252 usec, L: 0
0111 02:57:23.287420 (+ 25us) op_driver.cc:348] REPLICATION: starting
0111 02:57:23.288200 (+ 780us) log.cc:844] Serialized 356 byte log entry
0111 02:57:23.312232 (+ 24032us) op_driver.cc:464] REPLICATION: finished
0111 02:57:23.312529 (+ 297us) write_op.cc:301] APPLY: starting
0111 02:57:23.312603 (+ 74us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:23.312738 (+ 135us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:23.312891 (+ 153us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:23.313145 (+ 254us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:23.313323 (+ 178us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:23.313387 (+ 64us) write_op.cc:312] APPLY: finished
0111 02:57:23.313633 (+ 246us) log.cc:844] Serialized 23 byte log entry
0111 02:57:23.313784 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:23.313866 (+ 82us) write_op.cc:454] Released schema lock
0111 02:57:23.313899 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":6,"spinlock_wait_cycles":22451456,"child_traces":[["op",{"apply.queue_time_us":109,"apply.run_cpu_time_us":1552,"apply.run_wall_time_us":1551,"num_ops":1,"prepare.queue_time_us":117823,"prepare.run_cpu_time_us":2816,"prepare.run_wall_time_us":4244,"raft.queue_time_us":399,"raft.run_cpu_time_us":911,"raft.run_wall_time_us":913,"replication_time_us":24683}]]}
W20250111 02:57:23.638207 18986 consensus_peers.cc:487] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8128b3941d54ce2a72bf2ae762a774f. This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:23.666429 18622 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa86b1e6eca7403c9a9c52592b8fb636 has no consensus state
W20250111 02:57:23.678195 18629 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc7158a7fc624fa4bc816ac3721c610d consensus state has no leader
I20250111 02:57:23.532267 19325 ts_tablet_manager.cc:1918] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:23.752146 19325 log.cc:1198] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/29561e984e6d40fea032075f0531dd1f
I20250111 02:57:23.752957 19325 ts_tablet_manager.cc:1939] T 29561e984e6d40fea032075f0531dd1f P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:23.596968 19167 tablet_service.cc:1514] Processing DeleteTablet for tablet ef70b417568049f0b496f4d8e5f7e0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7ea78feee64340e39404cba0f1bf04db) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:23.597298 19169 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe4e60db4244a9a9696153b630c575b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:23.597693 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet c28958a1018344a48d18e6c0f03cd0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:17 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:23.646754 19695 raft_consensus.cc:491] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:23.759466 18627 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_dacea9000ff840659052c41c94a8bfaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:23.761096 18627 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dacea9000ff840659052c41c94a8bfaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:23.486222 19131 leader_election.cc:343] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: 0dbb63ab78f84995ae6d9fb2aae78dcf
I20250111 02:57:23.758692 19695 raft_consensus.cc:513] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.765197 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 148) took 1192 ms. Trace:
I20250111 02:57:23.769328 19325 rpcz_store.cc:276] 0111 02:57:22.572686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.620309 (+ 47623us) service_pool.cc:225] Handling call
0111 02:57:23.445397 (+825088us) ts_tablet_manager.cc:1145] Deleting tablet 29561e984e6d40fea032075f0531dd1f
0111 02:57:23.445439 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:23.491316 (+ 45877us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:23.532050 (+ 40734us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:23.765068 (+233018us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":824541}
I20250111 02:57:23.769876 19695 leader_election.cc:290] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:23.770867 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 29561e984e6d40fea032075f0531dd1f (table default.Table_acc960ebdd02430ab5aa41a4f75f8578 [id=dc8029d5413f4e838d330383cf0b6f70]) successfully deleted
I20250111 02:57:23.771538 19131 leader_election.cc:304] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
W20250111 02:57:23.772759 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 286592us
I20250111 02:57:23.773447 19695 raft_consensus.cc:3054] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:23.782850 19562 raft_consensus.cc:491] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:23.783370 19562 raft_consensus.cc:513] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.784286 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b9d14959df49a6bbd80ba9e9e57ee4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:23.785033 19054 raft_consensus.cc:2463] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:23.785892 19562 leader_election.cc:290] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:23.787446 19131 leader_election.cc:304] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:23.787726 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b9d14959df49a6bbd80ba9e9e57ee4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:23.788360 19197 raft_consensus.cc:2463] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:23.789680 18986 leader_election.cc:304] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:23.771899 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 147) took 1217 ms. Trace:
I20250111 02:57:23.790621 19325 rpcz_store.cc:276] 0111 02:57:22.554495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.623226 (+ 68731us) service_pool.cc:225] Handling call
0111 02:57:23.770366 (+1147140us) ts_tablet_manager.cc:1145] Deleting tablet 0267a92995ea460786dc26813b5895cf
0111 02:57:23.771133 (+ 767us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:23.771840 (+ 707us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1138196}
I20250111 02:57:23.792297 19325 tablet_replica.cc:331] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
W20250111 02:57:23.793295 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 0267a92995ea460786dc26813b5895cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0267a92995ea460786dc26813b5895cf already in progress: creating tablet
W20250111 02:57:23.793931 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 3052e9707de14e1f9e38236d56908ed3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3052e9707de14e1f9e38236d56908ed3 already in progress: creating tablet
I20250111 02:57:23.801347 19695 raft_consensus.cc:2743] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:23.802033 19325 raft_consensus.cc:2238] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:23.803907 19325 raft_consensus.cc:2267] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
W20250111 02:57:23.803663 18616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.126:45115, user_credentials={real_user=slave}} blocked reactor thread for 263701us
I20250111 02:57:23.808430 19481 raft_consensus.cc:2798] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:23.808904 19481 raft_consensus.cc:491] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:23.809303 19481 raft_consensus.cc:3054] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:23.813000 19321 tablet_bootstrap.cc:492] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:23.813453 19321 ts_tablet_manager.cc:1397] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.403s user 0.040s sys 0.072s
I20250111 02:57:23.807607 18627 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_02b5991089a34b0b8ee6039ccc092c93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:23.820865 18627 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02b5991089a34b0b8ee6039ccc092c93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:23.818346 19481 raft_consensus.cc:513] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.824394 19585 leader_election.cc:290] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:23.825253 19585 raft_consensus.cc:2798] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:23.825755 19585 raft_consensus.cc:491] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:23.826272 19585 raft_consensus.cc:3054] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:23.833722 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b9d14959df49a6bbd80ba9e9e57ee4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:23.839226 19481 leader_election.cc:290] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:23.837383 19325 ts_tablet_manager.cc:1905] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:23.841034 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b9d14959df49a6bbd80ba9e9e57ee4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:23.841531 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ef3f45e5784a29b3b32cc0de6bac84" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:23.842470 18908 raft_consensus.cc:3054] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:23.843400 19321 raft_consensus.cc:357] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.844076 19321 raft_consensus.cc:383] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:23.844336 19321 raft_consensus.cc:738] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
W20250111 02:57:23.848268 19133 leader_election.cc:343] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:23.849229 19321 consensus_queue.cc:260] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.857436 18623 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_7653ac6289d7467590ac03b43df1b8bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:23.862036 18623 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7653ac6289d7467590ac03b43df1b8bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:23.866890 19321 ts_tablet_manager.cc:1428] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.053s user 0.008s sys 0.000s
I20250111 02:57:23.871460 19321 tablet_bootstrap.cc:492] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:23.878517 18627 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_02a1e7d9f9094de5b4a61b79970284a7" } new_table_name: "default.Table_ef2e1cefc2fc447f8ea84f73c777f556" modify_external_catalogs: true
W20250111 02:57:23.879205 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 91900.8us
I20250111 02:57:23.884119 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ef3f45e5784a29b3b32cc0de6bac84" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:23.884788 19054 raft_consensus.cc:3054] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:23.882669 18625 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_a4c544f93b2a4ad5afd4aa78f3989b6f" } new_table_name: "default.Table_842f087ec75f45f2a6ba5652f3a0bde1" modify_external_catalogs: true
I20250111 02:57:23.892009 19322 tablet_bootstrap.cc:492] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:23.892594 19322 ts_tablet_manager.cc:1397] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 1.417s user 0.010s sys 0.009s
I20250111 02:57:23.894776 19321 tablet_bootstrap.cc:654] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:23.896226 19695 raft_consensus.cc:491] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:23.896665 19695 raft_consensus.cc:513] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.898577 19695 leader_election.cc:290] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:23.900349 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40fdec982cf643c68ef90798f4f65ad4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:23.901568 19133 leader_election.cc:343] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:23.902540 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_cce40438ec8e4527b01088af6b7b0981"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:23.904752 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cce40438ec8e4527b01088af6b7b0981 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:23.908304 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40fdec982cf643c68ef90798f4f65ad4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:23.908711 19054 raft_consensus.cc:2463] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:23.909008 19052 raft_consensus.cc:2463] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:23.914595 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40fdec982cf643c68ef90798f4f65ad4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:23.915402 19131 leader_election.cc:304] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
W20250111 02:57:23.916275 19692 log.cc:926] Time spent T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c: Append to log took a long time: real 0.337s user 0.001s sys 0.000s
W20250111 02:57:23.916322 18986 consensus_peers.cc:487] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fe4e60db4244a9a9696153b630c575b. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:23.916905 19695 raft_consensus.cc:2798] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:23.917342 19695 raft_consensus.cc:695] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:23.918184 18906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.18.20.66:58419 (request call id 72) took 1012 ms. Trace:
I20250111 02:57:23.918103 19695 consensus_queue.cc:237] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.918632 18906 rpcz_store.cc:276] 0111 02:57:22.905831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.907132 (+ 1301us) service_pool.cc:225] Handling call
0111 02:57:22.907622 (+ 490us) raft_consensus.cc:1466] Updating replica for 1 ops
0111 02:57:23.512335 (+604713us) raft_consensus.cc:1529] Early marking committed up to index 0
0111 02:57:23.512384 (+ 49us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0111 02:57:23.514060 (+ 1676us) log.cc:844] Serialized 26 byte log entry
0111 02:57:23.566506 (+ 52446us) raft_consensus.cc:1640] Marking committed up to 0
0111 02:57:23.582694 (+ 16188us) raft_consensus.cc:1692] Filling consensus response to leader.
0111 02:57:23.877492 (+294798us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0111 02:57:23.917907 (+ 40415us) raft_consensus.cc:1680] finished
0111 02:57:23.917936 (+ 29us) raft_consensus.cc:1686] UpdateReplicas() finished
0111 02:57:23.918101 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":56918,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":170,"thread_start_us":648739,"threads_started":2,"wal-append.queue_time_us":43814}
I20250111 02:57:23.924053 19131 leader_election.cc:304] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:23.931521 19696 raft_consensus.cc:2798] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:23.931900 19696 raft_consensus.cc:491] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:23.932191 19696 raft_consensus.cc:3054] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:23.939534 19696 raft_consensus.cc:513] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:23.943974 19602 consensus_meta.cc:287] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c: Time spent flushing consensus metadata: real 0.926s user 0.009s sys 0.003s
I20250111 02:57:23.945107 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40fdec982cf643c68ef90798f4f65ad4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:23.945816 19052 raft_consensus.cc:3054] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:23.944031 18905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40fdec982cf643c68ef90798f4f65ad4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:23.956005 19133 leader_election.cc:343] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:23.960156 19052 raft_consensus.cc:2463] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:23.961711 19696 leader_election.cc:290] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:23.970311 19325 ts_tablet_manager.cc:1918] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:23.950558 19602 raft_consensus.cc:513] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:23.987373 19325 log.cc:1198] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/0dbb63ab78f84995ae6d9fb2aae78dcf
I20250111 02:57:23.988366 19325 ts_tablet_manager.cc:1939] T 0dbb63ab78f84995ae6d9fb2aae78dcf P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:23.991402 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 0dbb63ab78f84995ae6d9fb2aae78dcf (table default.Table_154c30276c8d4a8e805dc2fb85867bbf [id=9d3552a625804f0a9350454f3b5dc2e8]) successfully deleted
I20250111 02:57:23.993804 19131 leader_election.cc:304] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:23.995713 19695 raft_consensus.cc:2798] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:23.996477 19695 raft_consensus.cc:695] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:23.997649 19695 consensus_queue.cc:237] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.005928 19321 tablet_bootstrap.cc:492] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:24.006558 19321 ts_tablet_manager.cc:1397] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.135s user 0.011s sys 0.000s
I20250111 02:57:24.010044 19321 raft_consensus.cc:357] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:24.017693 18625 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ea78feee64340e39404cba0f1bf04db consensus state has no leader
I20250111 02:57:23.988266 18909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.18.20.67:36199 (request call id 40) took 1111 ms. Trace:
I20250111 02:57:24.020579 18909 rpcz_store.cc:276] 0111 02:57:22.876480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.876711 (+ 231us) service_pool.cc:225] Handling call
0111 02:57:23.987935 (+1111224us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400180bd4. stack: 00007f02faaf7853 00007f02faaf7750 00007f02fa66c584 00000000004fcd24 00000000004fccb9 00007f03019f76d1 00007f0301a762ae 00007f0302a58c17 00007f0301226c14 00007f0301226ba2 00007f0301226ad1 00007f0301226a4c 00007f0301225d02 00007f02ff83fdfc 00007f02ff83f236 00007f02ff83ebdf
0111 02:57:23.988163 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2542297856}
I20250111 02:57:24.021833 19602 leader_election.cc:290] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:23.901556 19322 raft_consensus.cc:357] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.028424 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b9d14959df49a6bbd80ba9e9e57ee4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:24.029353 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet b26e5db463e94d919640a07a2f8574a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b26e5db463e94d919640a07a2f8574a8 already in progress: creating tablet
I20250111 02:57:24.029554 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b9d14959df49a6bbd80ba9e9e57ee4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:24.029752 18842 leader_election.cc:304] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters: d0cedab82b154d62b78809edc6f28bfa
W20250111 02:57:24.032341 18989 leader_election.cc:343] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:24.032850 18989 leader_election.cc:304] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f; no voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:24.035255 19533 raft_consensus.cc:2743] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:24.025499 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e8c72c99ac4c369957b571d74baffe" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:24.038436 19192 raft_consensus.cc:3054] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:57:24.039139 19562 consensus_queue.cc:1035] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:24.038100 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e8c72c99ac4c369957b571d74baffe" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:24.040660 19052 raft_consensus.cc:3054] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:57:24.042687 19677 consensus_queue.cc:1035] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250111 02:57:24.044552 18989 leader_election.cc:343] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:24.045661 19192 raft_consensus.cc:2463] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 2.
I20250111 02:57:24.030550 19325 tablet_replica.cc:331] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:24.047170 19325 raft_consensus.cc:2238] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:24.047641 19325 raft_consensus.cc:2267] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:24.048347 18841 leader_election.cc:304] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:24.049299 19325 ts_tablet_manager.cc:1905] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:24.049856 19321 raft_consensus.cc:383] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.050199 19321 raft_consensus.cc:738] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
W20250111 02:57:24.052075 18989 leader_election.cc:343] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:24.055588 19321 consensus_queue.cc:260] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.027309 19322 raft_consensus.cc:383] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.060616 19322 raft_consensus.cc:738] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:24.062031 19322 consensus_queue.cc:260] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.064188 19321 ts_tablet_manager.cc:1428] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.057s user 0.005s sys 0.001s
I20250111 02:57:24.065420 19133 leader_election.cc:304] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:24.065723 18623 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_1ab2b1f61ced4c97ad5e44f18b6e8699"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:24.066244 19133 raft_consensus.cc:2681] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250111 02:57:24.067303 18623 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ab2b1f61ced4c97ad5e44f18b6e8699 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:24.069417 18986 consensus_peers.cc:487] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29561e984e6d40fea032075f0531dd1f. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:24.073776 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_1431a031eea3496b9764a373a012a367"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:24.075629 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1431a031eea3496b9764a373a012a367 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:23.963418 18908 raft_consensus.cc:2463] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:24.049248 19052 raft_consensus.cc:2463] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 2.
I20250111 02:57:24.088248 19321 tablet_bootstrap.cc:492] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:24.095795 19545 raft_consensus.cc:2758] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250111 02:57:24.081840 19602 consensus_queue.cc:1035] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20250111 02:57:24.101639 19695 raft_consensus.cc:491] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:24.102144 19695 raft_consensus.cc:513] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.085636 19707 raft_consensus.cc:2798] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:57:24.104971 19707 raft_consensus.cc:695] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 2 LEADER]: Becoming Leader. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Running, Role: LEADER
I20250111 02:57:24.106300 19695 leader_election.cc:290] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:24.105780 19707 consensus_queue.cc:237] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:24.109966 18627 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48dd0bccf6854d37a7ef95b5ab06308e has no consensus state
I20250111 02:57:24.110109 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aa64b2de5746d1b12cc93a10c9d56e" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:24.111373 19133 leader_election.cc:343] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:24.111699 19309 thread.cc:635] rpc worker (service pool kudu.tserver.TabletCopyService) Time spent creating pthread: real 0.761s user 0.225s sys 0.412s
W20250111 02:57:24.112155 19309 thread.cc:601] rpc worker (service pool kudu.tserver.TabletCopyService) Time spent starting thread: real 0.762s user 0.225s sys 0.412s
I20250111 02:57:24.112365 18869 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:51078 (request call id 47) took 1399 ms. Trace:
W20250111 02:57:24.112826 19131 leader_election.cc:343] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: 92aa64b2de5746d1b12cc93a10c9d56e
I20250111 02:57:24.112715 18869 rpcz_store.cc:276] 0111 02:57:22.712795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.720406 (+ 7611us) service_pool.cc:225] Handling call
0111 02:57:23.583471 (+863065us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b5400160654. stack: 00007f02faaf7853 00007f02faaf7750 00007f02fa66c584 00000000004fcd24 00000000004fccb9 00007f03019f76d1 00007f0301a6c268 00007f0302a626bd 00007f0301300e14 00007f0301300da2 00007f0301300cd1 00007f0301300c4c 00007f03012fff02 00007f02ff83fdfc 00007f02ff83f236 00007f02ff83ebdf
0111 02:57:23.991700 (+408229us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b5400180bd4. stack: 00007f02faaf7853 00007f02faaf7750 00007f02fa66c584 00000000004fcd24 00000000004fccb9 00007f03019f76d1 00007f0301a6c268 00007f0302a626bd 00007f0301300e14 00007f0301300da2 00007f0301300cd1 00007f0301300c4c 00007f03012fff02 00007f02ff83fdfc 00007f02ff83f236 00007f02ff83ebdf
0111 02:57:24.043630 (+ 51930us) spinlock_profiling.cc:243] Waited 49 ms on lock 0x7b5400150254. stack: 00007f02faaf7853 00007f02faaf7750 00007f02fa66c584 00000000004fcd24 00000000004fccb9 00007f03019f76d1 00007f0301a6c268 00007f0302a626bd 00007f0301300e14 00007f0301300da2 00007f0301300cd1 00007f0301300c4c 00007f03012fff02 00007f02ff83fdfc 00007f02ff83f236 00007f02ff83ebdf
0111 02:57:24.112163 (+ 68533us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2271242112}
I20250111 02:57:24.113854 19131 leader_election.cc:304] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
W20250111 02:57:24.115329 18841 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 67107.1us
I20250111 02:57:24.116315 19696 raft_consensus.cc:2743] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:24.116858 19695 raft_consensus.cc:491] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:24.118233 19695 raft_consensus.cc:513] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.125442 19695 leader_election.cc:290] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:24.125659 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2377893bfc426fa00d5ddacd9c5961" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:24.126410 19054 raft_consensus.cc:2463] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:24.128232 19131 leader_election.cc:304] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:24.129952 18628 master_service.cc:946] ReplaceTablet: received request to replace tablet 5cf1ce65ccf54358b95665b9764fbdb0 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:24.134049 19695 raft_consensus.cc:2798] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:24.134471 19695 raft_consensus.cc:491] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:24.134835 19695 raft_consensus.cc:3054] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:24.135164 19322 ts_tablet_manager.cc:1428] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.241s user 0.005s sys 0.007s
I20250111 02:57:24.136447 19322 tablet_bootstrap.cc:492] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
W20250111 02:57:24.140132 19330 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.053s sys 0.102s
W20250111 02:57:24.140448 19330 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.588s user 0.053s sys 0.102s
I20250111 02:57:24.149549 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_52d947397a774f96a4fdb5087da4e19a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:24.152947 19330 tablet_bootstrap.cc:492] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
W20250111 02:57:24.151153 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52d947397a774f96a4fdb5087da4e19a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:24.153404 19330 ts_tablet_manager.cc:1397] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.616s user 0.060s sys 0.108s
I20250111 02:57:24.160802 19585 raft_consensus.cc:513] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.162184 19197 raft_consensus.cc:2388] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 54628fbcec2f4359aa9e26a734d4d29f in current term 1: Already voted for candidate d0cedab82b154d62b78809edc6f28bfa in this term.
I20250111 02:57:24.164692 18986 leader_election.cc:304] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f; no voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:24.165599 19481 raft_consensus.cc:2743] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250111 02:57:24.166885 19330 raft_consensus.cc:357] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.167670 19330 raft_consensus.cc:383] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.167924 19330 raft_consensus.cc:738] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:24.168692 19330 consensus_queue.cc:260] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.171855 19326 ts_tablet_manager.cc:1918] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:24.172325 19326 log.cc:1198] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/066c87476a854e169fd4f5f1cdbb49f7
I20250111 02:57:24.173166 19326 ts_tablet_manager.cc:1939] T 066c87476a854e169fd4f5f1cdbb49f7 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:24.187021 19321 tablet_bootstrap.cc:654] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:24.197976 19695 raft_consensus.cc:513] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.203514 19671 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:36190 (request call id 155) took 1043 ms. Trace:
I20250111 02:57:24.204821 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2377893bfc426fa00d5ddacd9c5961" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:24.205453 19052 raft_consensus.cc:3054] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:24.212442 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b9d14959df49a6bbd80ba9e9e57ee4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:24.213451 19054 raft_consensus.cc:2388] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d0cedab82b154d62b78809edc6f28bfa in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:24.215672 19695 leader_election.cc:290] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:24.216452 19585 leader_election.cc:290] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:24.219904 19330 ts_tablet_manager.cc:1428] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.066s user 0.006s sys 0.000s
I20250111 02:57:24.220911 19330 tablet_bootstrap.cc:492] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:24.225010 18628 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_ef2e1cefc2fc447f8ea84f73c777f556" } new_table_name: "default.Table_60c8bd1736a84a2d9fd52866d624704e" modify_external_catalogs: true
I20250111 02:57:24.231726 19325 ts_tablet_manager.cc:1918] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:24.232110 19325 log.cc:1198] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/ef70b417568049f0b496f4d8e5f7e0e4
I20250111 02:57:24.232761 19325 ts_tablet_manager.cc:1939] T ef70b417568049f0b496f4d8e5f7e0e4 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:24.235153 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet ef70b417568049f0b496f4d8e5f7e0e4 (table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5]) successfully deleted
I20250111 02:57:24.236625 19327 ts_tablet_manager.cc:1918] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250111 02:57:24.237259 19327 log.cc:1198] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/5baef8688ede42fca28cfb6b0bdeefa6
I20250111 02:57:24.238299 19327 ts_tablet_manager.cc:1939] T 5baef8688ede42fca28cfb6b0bdeefa6 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:24.239389 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 122) took 3111 ms. Trace:
I20250111 02:57:24.239629 19321 tablet_bootstrap.cc:492] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:24.240047 19321 ts_tablet_manager.cc:1397] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.152s user 0.012s sys 0.000s
I20250111 02:57:24.224226 19052 raft_consensus.cc:2463] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:24.241619 19131 leader_election.cc:304] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:24.230973 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 119) took 3709 ms. Trace:
I20250111 02:57:24.204140 19671 rpcz_store.cc:276] 0111 02:57:23.159797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:23.159955 (+ 158us) service_pool.cc:225] Handling call
0111 02:57:24.203406 (+1043451us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:23.161322 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0111 02:57:23.161780 (+ 458us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0111 02:57:23.161817 (+ 37us) alter_schema_op.cc:66] Acquired schema lock
0111 02:57:23.161928 (+ 111us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0111 02:57:23.162069 (+ 141us) alter_schema_op.cc:128] START. Timestamp: 1736564243162003
0111 02:57:23.162095 (+ 26us) op_driver.cc:348] REPLICATION: starting
0111 02:57:23.162675 (+ 580us) log.cc:844] Serialized 173 byte log entry
0111 02:57:23.225542 (+ 62867us) op_driver.cc:464] REPLICATION: finished
0111 02:57:23.486477 (+260935us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0111 02:57:23.535402 (+ 48925us) log.cc:844] Serialized 5 byte log entry
0111 02:57:24.097792 (+562390us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.098892 (+ 1100us) log.cc:844] Serialized 14 byte log entry
0111 02:57:24.203210 (+104318us) alter_schema_op.cc:73] Released schema lock
0111 02:57:24.203261 (+ 51us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":260661,"prepare.queue_time_us":238,"prepare.run_cpu_time_us":7017,"prepare.run_wall_time_us":22442,"replication_time_us":63281,"thread_start_us":129007,"threads_started":2,"wal-append.queue_time_us":10505}]]}
I20250111 02:57:24.244222 19326 rpcz_store.cc:276] 0111 02:57:20.521473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:20.521967 (+ 494us) service_pool.cc:225] Handling call
0111 02:57:23.019093 (+2497126us) ts_tablet_manager.cc:1145] Deleting tablet 066c87476a854e169fd4f5f1cdbb49f7
0111 02:57:23.019148 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:23.781045 (+761897us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.171572 (+390527us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.230911 (+ 59339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2496472}
I20250111 02:57:24.242784 19321 raft_consensus.cc:357] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.245777 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 066c87476a854e169fd4f5f1cdbb49f7 (table default.Table_211b4c873fa34d6b8d9f4c1a303d5b28 [id=6c02a835750c4952a48c4089e4fe120c]) successfully deleted
I20250111 02:57:24.245926 19321 raft_consensus.cc:383] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.246191 19321 raft_consensus.cc:738] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:24.246805 19326 tablet_replica.cc:331] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:24.247241 19585 raft_consensus.cc:2798] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:24.248113 19585 raft_consensus.cc:695] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:24.249008 19585 consensus_queue.cc:237] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:24.250254 18628 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48dd0bccf6854d37a7ef95b5ab06308e has no consensus state
I20250111 02:57:24.249337 19326 raft_consensus.cc:2238] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:24.258937 19326 raft_consensus.cc:2267] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:24.261318 19326 ts_tablet_manager.cc:1905] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:24.262573 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 9fe4e60db4244a9a9696153b630c575b because the tablet was not found. No further retry: Not found: Tablet not found: 9fe4e60db4244a9a9696153b630c575b
I20250111 02:57:24.253741 19321 consensus_queue.cc:260] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.266589 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_f9181ca352384593b93119fb3e0cd812"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:24.267984 19327 rpcz_store.cc:276] 0111 02:57:21.128100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.147269 (+ 19169us) service_pool.cc:225] Handling call
0111 02:57:23.039070 (+1891801us) ts_tablet_manager.cc:1145] Deleting tablet 5baef8688ede42fca28cfb6b0bdeefa6
0111 02:57:23.039149 (+ 79us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:24.053608 (+1014459us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.236386 (+182778us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.239313 (+ 2927us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1592168}
I20250111 02:57:24.268612 19325 tablet_replica.cc:331] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
W20250111 02:57:24.268620 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9181ca352384593b93119fb3e0cd812 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:24.269477 19327 tablet_replica.cc:331] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:24.279999 19321 ts_tablet_manager.cc:1428] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20250111 02:57:24.280987 19325 raft_consensus.cc:2238] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:24.283897 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 5baef8688ede42fca28cfb6b0bdeefa6 (table default.Table_7f4578da9f5e418c9d67a535471b45cd [id=154c37dc0dba44bdab5e0315756dc7cf]) successfully deleted
I20250111 02:57:24.287786 19325 raft_consensus.cc:2267] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:24.289516 18627 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]). Expected version 2 got 0
I20250111 02:57:24.289999 19326 ts_tablet_manager.cc:1918] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:24.290323 19326 log.cc:1198] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/5118d4b9cfbb49e8af6af56180858336
I20250111 02:57:24.290483 18627 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet f1802e7349934fbb9d5f7eda37c3df20 (table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0]). Expected version 4 got 0
I20250111 02:57:24.291011 19326 ts_tablet_manager.cc:1939] T 5118d4b9cfbb49e8af6af56180858336 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:24.293006 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 120) took 3771 ms. Trace:
I20250111 02:57:24.293319 19326 rpcz_store.cc:276] 0111 02:57:20.521827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:20.522433 (+ 606us) service_pool.cc:225] Handling call
0111 02:57:24.246615 (+3724182us) ts_tablet_manager.cc:1145] Deleting tablet 5118d4b9cfbb49e8af6af56180858336
0111 02:57:24.246665 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:24.276096 (+ 29431us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.289789 (+ 13693us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.292939 (+ 3150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3723660}
I20250111 02:57:24.294576 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 5118d4b9cfbb49e8af6af56180858336 (table default.Table_39701f385f884aac9233a842fcaf93be [id=bb83163ef256406da8650e72eadf7f6c]) successfully deleted
I20250111 02:57:24.302350 18622 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]). Expected version 2 got 0
I20250111 02:57:24.302731 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 123) took 3635 ms. Trace:
I20250111 02:57:24.303143 18624 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 0ef86e1fd412484db83e7d39c9059a90
I20250111 02:57:24.303033 19326 rpcz_store.cc:276] 0111 02:57:20.667357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:20.670809 (+ 3452us) service_pool.cc:225] Handling call
0111 02:57:24.296339 (+3625530us) ts_tablet_manager.cc:1145] Deleting tablet 5cf1ce65ccf54358b95665b9764fbdb0
0111 02:57:24.296381 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:24.302669 (+ 6288us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3624903}
W20250111 02:57:24.304602 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 5cf1ce65ccf54358b95665b9764fbdb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cf1ce65ccf54358b95665b9764fbdb0 already in progress: creating tablet
I20250111 02:57:24.305425 19326 tablet_replica.cc:331] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:24.306157 19326 raft_consensus.cc:2238] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:24.306587 19326 raft_consensus.cc:2267] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:24.309597 19174 tablet_service.cc:1467] Processing CreateTablet for tablet 7145efe5c0094b70be2cd6481b770248 (DEFAULT_TABLE table=default.Table_7ee7d20cfb7e4985bdd8539d15e354fe [id=ff66cd83751a45a09d8012cc8a9f1758]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:24.311028 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7145efe5c0094b70be2cd6481b770248. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:24.310817 19029 tablet_service.cc:1467] Processing CreateTablet for tablet 7145efe5c0094b70be2cd6481b770248 (DEFAULT_TABLE table=default.Table_7ee7d20cfb7e4985bdd8539d15e354fe [id=ff66cd83751a45a09d8012cc8a9f1758]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:24.312422 19029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7145efe5c0094b70be2cd6481b770248. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:24.317041 19033 tablet_service.cc:1467] Processing CreateTablet for tablet df1870cceafe463b8d024e8b065850e0 (DEFAULT_TABLE table=default.Table_aea3cbca3247424c9a12e310135fe0b9 [id=c151ba8ccc454372b4ad7621b6bc7aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:24.318065 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df1870cceafe463b8d024e8b065850e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:24.318424 19177 tablet_service.cc:1467] Processing CreateTablet for tablet 2873325cece64db6a673d4fe08eb7409 (DEFAULT_TABLE table=default.Table_c9582d98645645f3aa80a65f7652cba8 [id=2d5d8f8a7ade443f91a7da34add25a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:24.319710 19024 tablet_service.cc:1467] Processing CreateTablet for tablet 42230a0bc94249fda91cf79fba48f024 (DEFAULT_TABLE table=default.Table_d693cd14e5074b6b8ef78428cb948e48 [id=8677d1f5d44c4967873055efe4d4be40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:24.322839 19330 tablet_bootstrap.cc:654] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:24.326067 18627 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_28852db6fe0d4d5bae2db5ad7fa3ac63" } new_table_name: "default.Table_94ff268a8e3942ae997a5d44c55b4f43" modify_external_catalogs: true
I20250111 02:57:24.327088 19024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42230a0bc94249fda91cf79fba48f024. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:24.334430 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_85a0376bcaf24d0081e780e8c19764b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:24.336433 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85a0376bcaf24d0081e780e8c19764b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:24.346231 19326 ts_tablet_manager.cc:1905] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:24.355031 19028 tablet_service.cc:1467] Processing CreateTablet for tablet 2873325cece64db6a673d4fe08eb7409 (DEFAULT_TABLE table=default.Table_c9582d98645645f3aa80a65f7652cba8 [id=2d5d8f8a7ade443f91a7da34add25a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:24.357250 18624 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47522 (request call id 203) took 1034 ms. Trace:
I20250111 02:57:24.357975 19028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2873325cece64db6a673d4fe08eb7409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:24.360864 19023 tablet_service.cc:1467] Processing CreateTablet for tablet 2009d6f54e2842e8be3af2cc9147dc2c (DEFAULT_TABLE table=default.Table_24e5a7f819cc45dbb7efbd7286ad7005 [id=c04c27fb52054de185e1f3075f89a59a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:24.362164 19023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2009d6f54e2842e8be3af2cc9147dc2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:24.363337 19026 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ef86e1fd412484db83e7d39c9059a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:23 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:24.373636 18870 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:51078 (request call id 48) took 1384 ms. Trace:
I20250111 02:57:24.374050 18870 rpcz_store.cc:276] 0111 02:57:22.989386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.989612 (+ 226us) service_pool.cc:225] Handling call
0111 02:57:23.619102 (+629490us) spinlock_profiling.cc:243] Waited 617 ms on lock 0x7b5400160654. stack: 00007f02faaf7853 00007f02faaf7750 00007f02fa66c584 00000000004fcd24 00000000004fccb9 00007f03019f76d1 00007f0301a6c268 00007f0302a626bd 00007f0301300e14 00007f0301300da2 00007f0301300cd1 00007f0301300c4c 00007f03012fff02 00007f02ff83fdfc 00007f02ff83f236 00007f02ff83ebdf
0111 02:57:24.090740 (+471638us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b5400190d54. stack: 00007f02faaf7853 00007f02faaf7750 00007f02fa66c584 00000000004fcd24 00000000004fccb9 00007f03019f76d1 00007f0301a6c268 00007f0302a626bd 00007f0301300e14 00007f0301300da2 00007f0301300cd1 00007f0301300c4c 00007f03012fff02 00007f02ff83fdfc 00007f02ff83f236 00007f02ff83ebdf
0111 02:57:24.229927 (+139187us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b5400150254. stack: 00007f02faaf7853 00007f02faaf7750 00007f02fa66c584 00000000004fcd24 00000000004fccb9 00007f03019f76d1 00007f0301a6c268 00007f0302a626bd 00007f0301300e14 00007f0301300da2 00007f0301300cd1 00007f0301300c4c 00007f03012fff02 00007f02ff83fdfc 00007f02ff83f236 00007f02ff83ebdf
0111 02:57:24.373547 (+143620us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2166092416}
I20250111 02:57:24.384446 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_97e52a18fbd44f189d46ba9b26cd9033"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:24.318984 19169 tablet_service.cc:1467] Processing CreateTablet for tablet df1870cceafe463b8d024e8b065850e0 (DEFAULT_TABLE table=default.Table_aea3cbca3247424c9a12e310135fe0b9 [id=c151ba8ccc454372b4ad7621b6bc7aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:24.385571 19325 ts_tablet_manager.cc:1905] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:24.357676 18624 rpcz_store.cc:276] 0111 02:57:23.323261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:23.324027 (+ 766us) service_pool.cc:225] Handling call
0111 02:57:23.325377 (+ 1350us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0111 02:57:23.543907 (+218530us) catalog_manager.cc:2753] Modifying in-memory table state
0111 02:57:23.544102 (+ 195us) catalog_manager.cc:2762] Locking tablets
0111 02:57:24.276631 (+732529us) catalog_manager.cc:2776] Removing table and tablets from system table
0111 02:57:24.302610 (+ 25979us) catalog_manager.cc:2794] Removing table from by-name map
0111 02:57:24.302887 (+ 277us) catalog_manager.cc:2810] Committing in-memory state
0111 02:57:24.303029 (+ 142us) catalog_manager.cc:2814] Aborting table tasks
0111 02:57:24.357166 (+ 54137us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:24.285873 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:24.286251 (+ 378us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:24.286287 (+ 36us) write_op.cc:435] Acquired schema lock
0111 02:57:24.286309 (+ 22us) tablet.cc:659] Decoding operations
0111 02:57:24.286624 (+ 315us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:24.286664 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:24.286688 (+ 24us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:24.286938 (+ 250us) tablet.cc:698] Row locks acquired
0111 02:57:24.286951 (+ 13us) write_op.cc:260] PREPARE: finished
0111 02:57:24.287084 (+ 133us) write_op.cc:270] Start()
0111 02:57:24.287169 (+ 85us) write_op.cc:276] Timestamp: P: 1736564244287033 usec, L: 0
0111 02:57:24.287192 (+ 23us) op_driver.cc:348] REPLICATION: starting
0111 02:57:24.287744 (+ 552us) log.cc:844] Serialized 728 byte log entry
0111 02:57:24.300103 (+ 12359us) op_driver.cc:464] REPLICATION: finished
0111 02:57:24.300360 (+ 257us) write_op.cc:301] APPLY: starting
0111 02:57:24.300420 (+ 60us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:24.300547 (+ 127us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:24.300653 (+ 106us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:24.300898 (+ 245us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:24.301025 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0111 02:57:24.301066 (+ 41us) write_op.cc:312] APPLY: finished
0111 02:57:24.301241 (+ 175us) log.cc:844] Serialized 29 byte log entry
0111 02:57:24.301385 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:24.301448 (+ 63us) write_op.cc:454] Released schema lock
0111 02:57:24.301492 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:24.301704 (+ 212us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:24.301726 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":88,"apply.run_cpu_time_us":1313,"apply.run_wall_time_us":1313,"num_ops":2,"prepare.queue_time_us":6846,"prepare.run_cpu_time_us":2270,"prepare.run_wall_time_us":9469,"raft.queue_time_us":165,"raft.run_cpu_time_us":714,"raft.run_wall_time_us":716,"replication_time_us":12815}]]}
I20250111 02:57:24.318424 19167 tablet_service.cc:1467] Processing CreateTablet for tablet 42230a0bc94249fda91cf79fba48f024 (DEFAULT_TABLE table=default.Table_d693cd14e5074b6b8ef78428cb948e48 [id=8677d1f5d44c4967873055efe4d4be40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:24.319664 19177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2873325cece64db6a673d4fe08eb7409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:24.355926 19168 tablet_service.cc:1467] Processing CreateTablet for tablet 2009d6f54e2842e8be3af2cc9147dc2c (DEFAULT_TABLE table=default.Table_24e5a7f819cc45dbb7efbd7286ad7005 [id=c04c27fb52054de185e1f3075f89a59a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:24.388715 19322 tablet_bootstrap.cc:654] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:24.357939 19321 tablet_bootstrap.cc:492] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:24.396668 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ef86e1fd412484db83e7d39c9059a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:23 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:24.397722 19168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2009d6f54e2842e8be3af2cc9147dc2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:24.348205 19173 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ef86e1fd412484db83e7d39c9059a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:23 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:24.410583 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df1870cceafe463b8d024e8b065850e0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:24.386355 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97e52a18fbd44f189d46ba9b26cd9033 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:24.396116 18623 master_service.cc:946] ReplaceTablet: received request to replace tablet 74592f80afab4513a9eb2eac351c4de5 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:24.414620 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42230a0bc94249fda91cf79fba48f024. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:24.416417 19707 raft_consensus.cc:491] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:24.417548 19707 raft_consensus.cc:513] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.420195 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2377893bfc426fa00d5ddacd9c5961" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:24.421021 18910 raft_consensus.cc:2463] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:24.421325 18905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2377893bfc426fa00d5ddacd9c5961" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:24.422091 18905 raft_consensus.cc:3054] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:24.422397 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b9d14959df49a6bbd80ba9e9e57ee4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:24.425061 18882 tablet_service.cc:1467] Processing CreateTablet for tablet 7145efe5c0094b70be2cd6481b770248 (DEFAULT_TABLE table=default.Table_7ee7d20cfb7e4985bdd8539d15e354fe [id=ff66cd83751a45a09d8012cc8a9f1758]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:24.426132 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7145efe5c0094b70be2cd6481b770248. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:24.427286 19707 leader_election.cc:290] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:24.403465 18842 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 60520us
W20250111 02:57:24.432574 19133 leader_election.cc:343] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:24.433312 19133 leader_election.cc:304] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:24.434504 19585 raft_consensus.cc:2743] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250111 02:57:24.446352 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_6ad0b827920c42c68113310190019df1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:24.447997 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ad0b827920c42c68113310190019df1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:24.468405 18626 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet b8e8c72c99ac4c369957b571d74baffe
I20250111 02:57:24.469520 19330 tablet_bootstrap.cc:492] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:24.470026 19330 ts_tablet_manager.cc:1397] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.249s user 0.013s sys 0.001s
I20250111 02:57:24.470094 18626 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47522 (request call id 206) took 1076 ms. Trace:
I20250111 02:57:24.471810 19585 raft_consensus.cc:491] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:24.472337 19585 raft_consensus.cc:513] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.472834 19330 raft_consensus.cc:357] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.473488 19330 raft_consensus.cc:383] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.473759 19330 raft_consensus.cc:738] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:24.474433 19330 consensus_queue.cc:260] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.470408 18626 rpcz_store.cc:276] 0111 02:57:23.394116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:23.394500 (+ 384us) service_pool.cc:225] Handling call
0111 02:57:23.395924 (+ 1424us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0111 02:57:23.557984 (+162060us) catalog_manager.cc:2753] Modifying in-memory table state
0111 02:57:23.558178 (+ 194us) catalog_manager.cc:2762] Locking tablets
0111 02:57:24.285899 (+727721us) catalog_manager.cc:2776] Removing table and tablets from system table
0111 02:57:24.467985 (+182086us) catalog_manager.cc:2794] Removing table from by-name map
0111 02:57:24.468198 (+ 213us) catalog_manager.cc:2810] Committing in-memory state
0111 02:57:24.468313 (+ 115us) catalog_manager.cc:2814] Aborting table tasks
0111 02:57:24.470031 (+ 1718us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:24.299386 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:24.387411 (+ 88025us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:24.387436 (+ 25us) write_op.cc:435] Acquired schema lock
0111 02:57:24.387450 (+ 14us) tablet.cc:659] Decoding operations
0111 02:57:24.405919 (+ 18469us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:24.405964 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:24.405986 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:24.406286 (+ 300us) tablet.cc:698] Row locks acquired
0111 02:57:24.406300 (+ 14us) write_op.cc:260] PREPARE: finished
0111 02:57:24.406426 (+ 126us) write_op.cc:270] Start()
0111 02:57:24.406517 (+ 91us) write_op.cc:276] Timestamp: P: 1736564244406402 usec, L: 0
0111 02:57:24.406531 (+ 14us) op_driver.cc:348] REPLICATION: starting
0111 02:57:24.443697 (+ 37166us) log.cc:844] Serialized 759 byte log entry
0111 02:57:24.465498 (+ 21801us) op_driver.cc:464] REPLICATION: finished
0111 02:57:24.465992 (+ 494us) write_op.cc:301] APPLY: starting
0111 02:57:24.466055 (+ 63us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:24.466187 (+ 132us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:24.466316 (+ 129us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:24.466668 (+ 352us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:24.466857 (+ 189us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0111 02:57:24.466912 (+ 55us) write_op.cc:312] APPLY: finished
0111 02:57:24.467182 (+ 270us) log.cc:844] Serialized 29 byte log entry
0111 02:57:24.467414 (+ 232us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:24.467535 (+ 121us) write_op.cc:454] Released schema lock
0111 02:57:24.467579 (+ 44us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":297,"apply.run_cpu_time_us":1827,"apply.run_wall_time_us":1826,"num_ops":2,"prepare.queue_time_us":11642,"raft.queue_time_us":319,"raft.run_cpu_time_us":772,"raft.run_wall_time_us":773,"replication_time_us":58829}]]}
I20250111 02:57:24.476661 19330 ts_tablet_manager.cc:1428] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20250111 02:57:24.477253 19026 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e8c72c99ac4c369957b571d74baffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:23 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:24.481807 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0267a92995ea460786dc26813b5895cf" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:24.482570 19052 raft_consensus.cc:2463] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:24.485816 19131 leader_election.cc:304] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:24.486912 19695 raft_consensus.cc:2798] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:24.487337 19695 raft_consensus.cc:491] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:24.487740 19695 raft_consensus.cc:3054] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:24.489562 19173 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e8c72c99ac4c369957b571d74baffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:23 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:24.491428 19321 tablet_bootstrap.cc:654] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:24.493041 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0267a92995ea460786dc26813b5895cf" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:24.496246 18624 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_3e9cad78bc1447c8806da38d51f85778" } new_table_name: "default.Table_4c9aebb06d554107a5448d968fdb0436" modify_external_catalogs: true
I20250111 02:57:24.474874 19585 leader_election.cc:290] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
W20250111 02:57:24.502956 19133 leader_election.cc:343] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:24.508316 18884 tablet_service.cc:1467] Processing CreateTablet for tablet df1870cceafe463b8d024e8b065850e0 (DEFAULT_TABLE table=default.Table_aea3cbca3247424c9a12e310135fe0b9 [id=c151ba8ccc454372b4ad7621b6bc7aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:24.509795 18884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df1870cceafe463b8d024e8b065850e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:24.509477 18881 tablet_service.cc:1467] Processing CreateTablet for tablet 2009d6f54e2842e8be3af2cc9147dc2c (DEFAULT_TABLE table=default.Table_24e5a7f819cc45dbb7efbd7286ad7005 [id=c04c27fb52054de185e1f3075f89a59a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:24.510805 18890 tablet_service.cc:1467] Processing CreateTablet for tablet 42230a0bc94249fda91cf79fba48f024 (DEFAULT_TABLE table=default.Table_d693cd14e5074b6b8ef78428cb948e48 [id=8677d1f5d44c4967873055efe4d4be40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:24.511348 18879 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e8c72c99ac4c369957b571d74baffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:23 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:24.510893 18881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2009d6f54e2842e8be3af2cc9147dc2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:24.511365 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2377893bfc426fa00d5ddacd9c5961" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:24.512645 18880 tablet_service.cc:1467] Processing CreateTablet for tablet 2873325cece64db6a673d4fe08eb7409 (DEFAULT_TABLE table=default.Table_c9582d98645645f3aa80a65f7652cba8 [id=2d5d8f8a7ade443f91a7da34add25a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:24.516252 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2377893bfc426fa00d5ddacd9c5961" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:24.517207 19052 raft_consensus.cc:2388] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate d0cedab82b154d62b78809edc6f28bfa in this term.
I20250111 02:57:24.518558 18842 leader_election.cc:304] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:24.519420 19695 raft_consensus.cc:513] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.520002 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2873325cece64db6a673d4fe08eb7409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:24.522711 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0267a92995ea460786dc26813b5895cf" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:24.523308 19052 raft_consensus.cc:3054] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:24.532446 19052 raft_consensus.cc:2463] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:24.537381 19131 leader_election.cc:304] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:24.539974 19585 raft_consensus.cc:2798] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:24.540480 19585 raft_consensus.cc:695] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:24.541453 19585 consensus_queue.cc:237] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.551959 19695 leader_election.cc:290] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:24.549532 19330 tablet_bootstrap.cc:492] T 9192160e931445f1acd207604192f759 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:24.506835 18905 raft_consensus.cc:2463] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:24.570758 19321 tablet_bootstrap.cc:492] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:24.571192 19321 ts_tablet_manager.cc:1397] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.256s user 0.012s sys 0.002s
I20250111 02:57:24.574630 19330 tablet_bootstrap.cc:654] T 9192160e931445f1acd207604192f759 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:24.575428 18842 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 57017.5us
I20250111 02:57:24.574569 19321 raft_consensus.cc:357] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.576406 19321 raft_consensus.cc:383] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.576680 19321 raft_consensus.cc:738] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:24.577914 19650 tablet.cc:1721] T f1802e7349934fbb9d5f7eda37c3df20 P 8221604733fe4fc0a1fc32de8d6af71c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250111 02:57:24.579370 18890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42230a0bc94249fda91cf79fba48f024. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:24.570104 19326 ts_tablet_manager.cc:1918] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:24.579972 19326 log.cc:1198] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/ede92343d9b54c29b8173a5eafcf68c6
I20250111 02:57:24.580766 19326 ts_tablet_manager.cc:1939] T ede92343d9b54c29b8173a5eafcf68c6 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:24.581544 19325 ts_tablet_manager.cc:1918] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:24.581908 19325 log.cc:1198] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/c28958a1018344a48d18e6c0f03cd0a3
I20250111 02:57:24.582602 19325 ts_tablet_manager.cc:1939] T c28958a1018344a48d18e6c0f03cd0a3 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:24.583628 19706 raft_consensus.cc:2743] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:24.591684 18905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0267a92995ea460786dc26813b5895cf" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:24.593778 19133 leader_election.cc:343] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:24.594460 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 129) took 3432 ms. Trace:
I20250111 02:57:24.594839 19326 rpcz_store.cc:276] 0111 02:57:21.162286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.162436 (+ 150us) service_pool.cc:225] Handling call
0111 02:57:24.305265 (+3142829us) ts_tablet_manager.cc:1145] Deleting tablet ede92343d9b54c29b8173a5eafcf68c6
0111 02:57:24.305306 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:24.522621 (+217315us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.569869 (+ 47248us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.594390 (+ 24521us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3141978}
I20250111 02:57:24.607439 19321 consensus_queue.cc:260] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.620338 19321 ts_tablet_manager.cc:1428] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.049s user 0.005s sys 0.001s
I20250111 02:57:24.623922 19326 tablet_replica.cc:331] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:24.626122 19321 tablet_bootstrap.cc:492] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:24.626144 19327 raft_consensus.cc:2238] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:24.626713 19327 raft_consensus.cc:2267] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:24.628093 19326 raft_consensus.cc:2238] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:24.629741 19326 raft_consensus.cc:2267] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:24.630142 19327 ts_tablet_manager.cc:1905] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:24.632436 19326 ts_tablet_manager.cc:1905] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:24.641366 19330 tablet_bootstrap.cc:492] T 9192160e931445f1acd207604192f759 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:24.641968 19330 ts_tablet_manager.cc:1397] T 9192160e931445f1acd207604192f759 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.093s user 0.005s sys 0.012s
I20250111 02:57:24.645785 19330 raft_consensus.cc:357] T 9192160e931445f1acd207604192f759 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.652345 19330 raft_consensus.cc:383] T 9192160e931445f1acd207604192f759 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.653170 19330 raft_consensus.cc:738] T 9192160e931445f1acd207604192f759 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:24.656317 19330 consensus_queue.cc:260] T 9192160e931445f1acd207604192f759 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.658984 19327 ts_tablet_manager.cc:1918] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:24.545413 18626 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_28d700d6f8444cfeab3dc52dd2bec17f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:24.584299 18617 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet e4bd42ad66064bcaa043746081f1c759 (table default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet e4bd42ad66064bcaa043746081f1c759 consensus state has no leader
I20250111 02:57:24.623230 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet ede92343d9b54c29b8173a5eafcf68c6 (table default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]) successfully deleted
I20250111 02:57:24.662660 19321 tablet_bootstrap.cc:654] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:24.664283 18621 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_d693cd14e5074b6b8ef78428cb948e48" } modify_external_catalogs: true
I20250111 02:57:24.664882 18621 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_d693cd14e5074b6b8ef78428cb948e48" } modify_external_catalogs: true
I20250111 02:57:24.667780 19327 log.cc:1198] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/9fe4e60db4244a9a9696153b630c575b
I20250111 02:57:24.668677 19327 ts_tablet_manager.cc:1939] T 9fe4e60db4244a9a9696153b630c575b P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
W20250111 02:57:24.668175 18626 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28d700d6f8444cfeab3dc52dd2bec17f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:24.669602 19330 ts_tablet_manager.cc:1428] T 9192160e931445f1acd207604192f759 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.027s user 0.003s sys 0.003s
I20250111 02:57:24.669883 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 123) took 3489 ms. Trace:
I20250111 02:57:24.670635 19327 rpcz_store.cc:276] 0111 02:57:21.180547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.180733 (+ 186us) service_pool.cc:225] Handling call
0111 02:57:24.269328 (+3088595us) ts_tablet_manager.cc:1145] Deleting tablet 9fe4e60db4244a9a9696153b630c575b
0111 02:57:24.269371 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:24.644048 (+374677us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.658736 (+ 14688us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.669775 (+ 11039us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":355796,"tablet-delete.queue_time_us":3087837}
I20250111 02:57:24.671378 19330 tablet_bootstrap.cc:492] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:24.672260 19327 tablet_replica.cc:331] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:24.672994 19327 raft_consensus.cc:2238] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:24.673563 19327 raft_consensus.cc:2267] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:24.675769 19327 ts_tablet_manager.cc:1905] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:24.678957 19330 tablet_bootstrap.cc:654] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:24.683912 19326 ts_tablet_manager.cc:1918] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:24.684397 19326 log.cc:1198] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/616655aa75824fffa46f81c4f95d5159
I20250111 02:57:24.685096 19326 ts_tablet_manager.cc:1939] T 616655aa75824fffa46f81c4f95d5159 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:24.688596 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 130) took 3525 ms. Trace:
I20250111 02:57:24.689070 19326 rpcz_store.cc:276] 0111 02:57:21.163343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.163473 (+ 130us) service_pool.cc:225] Handling call
0111 02:57:24.623766 (+3460293us) ts_tablet_manager.cc:1145] Deleting tablet 616655aa75824fffa46f81c4f95d5159
0111 02:57:24.623808 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:24.659223 (+ 35415us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.683687 (+ 24464us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.688529 (+ 4842us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3459730}
I20250111 02:57:24.689683 19321 tablet_bootstrap.cc:492] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:24.690613 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 616655aa75824fffa46f81c4f95d5159 (table default.Table_a58de727de864b9a8ecab09048b72fb7 [id=581224f40f0c4f03a9224f06d660c883]) successfully deleted
I20250111 02:57:24.691722 19321 ts_tablet_manager.cc:1397] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.066s user 0.013s sys 0.000s
I20250111 02:57:24.691771 19326 tablet_replica.cc:331] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:24.694236 19321 raft_consensus.cc:357] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:24.695164 19326 raft_consensus.cc:2238] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:24.695739 19326 raft_consensus.cc:2267] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:24.698488 19326 ts_tablet_manager.cc:1905] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:24.694900 19321 raft_consensus.cc:383] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.710405 19330 tablet_bootstrap.cc:492] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:24.710680 19321 raft_consensus.cc:738] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:24.711138 19330 ts_tablet_manager.cc:1397] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.005s
I20250111 02:57:24.713021 19325 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 155) took 1116 ms. Trace:
I20250111 02:57:24.713413 19325 rpcz_store.cc:276] 0111 02:57:23.597114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:23.597587 (+ 473us) service_pool.cc:225] Handling call
0111 02:57:24.267643 (+670056us) ts_tablet_manager.cc:1145] Deleting tablet c28958a1018344a48d18e6c0f03cd0a3
0111 02:57:24.267687 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:24.482351 (+214664us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.581295 (+ 98944us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.711371 (+130076us) spinlock_profiling.cc:243] Waited 89.2 ms on lock 0x7b20000b0530. stack: 00007fe197db8853 00007fe197db8750 00007fe19792d584 00000000004fcd24 00000000004fccb9 00007fe19fc7f581 00007fe19fcb0743 00007fe19fcb7ec2 00007fe19fcb7e41 00007fe19fcb7dd1 00007fe19fcb7d91 00007fe19fcb6afd 000000000059ea04 000000000059e839 00007fe197d3dfaf 00007fe19f156501
0111 02:57:24.712949 (+ 1578us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11598,"spinlock_wait_cycles":205192576,"tablet-delete.queue_time_us":509111}
I20250111 02:57:24.714140 19330 raft_consensus.cc:357] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:24.715170 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet c28958a1018344a48d18e6c0f03cd0a3 (table default.Table_7bf5032295a34c379d00c2d9ad7e417f [id=f6f9076d18654f339a6feac0c468e265]) successfully deleted
I20250111 02:57:24.715526 19321 consensus_queue.cc:260] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:24.717290 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 0ef86e1fd412484db83e7d39c9059a90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ef86e1fd412484db83e7d39c9059a90 already in progress: creating tablet
I20250111 02:57:24.718240 19325 tablet_replica.cc:331] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:24.719034 19325 raft_consensus.cc:2238] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:24.719594 19325 raft_consensus.cc:2267] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:24.720606 19330 raft_consensus.cc:383] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.720904 19330 raft_consensus.cc:738] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:24.721596 19330 consensus_queue.cc:260] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:24.722965 19325 ts_tablet_manager.cc:1905] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:24.723824 19330 ts_tablet_manager.cc:1428] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20250111 02:57:24.723920 19321 ts_tablet_manager.cc:1428] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20250111 02:57:24.725566 19330 tablet_bootstrap.cc:492] T a983a670c2104398ac1a159ac3a1da7c P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:24.726626 19321 tablet_bootstrap.cc:492] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:24.739127 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 9fe4e60db4244a9a9696153b630c575b (table default.Table_064d4893c74f4870bc45572dd481c63c [id=b0a0ff2f4d554503a18d05e0ff51402d]) successfully deleted
I20250111 02:57:24.742103 18622 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0] alter complete (version 4)
I20250111 02:57:24.744848 19326 ts_tablet_manager.cc:1918] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:24.747493 19327 ts_tablet_manager.cc:1918] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:24.747980 19327 log.cc:1198] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/d8128b3941d54ce2a72bf2ae762a774f
I20250111 02:57:24.748752 19327 ts_tablet_manager.cc:1939] T d8128b3941d54ce2a72bf2ae762a774f P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:24.749539 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 124) took 3564 ms. Trace:
I20250111 02:57:24.745220 19326 log.cc:1198] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/b54311f4269d4575b8998d7f545000e7
I20250111 02:57:24.751188 19326 ts_tablet_manager.cc:1939] T b54311f4269d4575b8998d7f545000e7 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:24.750656 19327 rpcz_store.cc:276] 0111 02:57:21.185528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.185665 (+ 137us) service_pool.cc:225] Handling call
0111 02:57:24.672102 (+3486437us) ts_tablet_manager.cc:1145] Deleting tablet d8128b3941d54ce2a72bf2ae762a774f
0111 02:57:24.672147 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:24.739661 (+ 67514us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.747231 (+ 7570us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.749485 (+ 2254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3485281}
I20250111 02:57:24.752041 19327 tablet_replica.cc:331] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:24.751498 19028 tablet_service.cc:1514] Processing DeleteTablet for tablet 5118d4b9cfbb49e8af6af56180858336 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:24.753613 19321 tablet_bootstrap.cc:654] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:24.753741 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet d8128b3941d54ce2a72bf2ae762a774f (table default.Table_8b6452ba06af45bab320b50842e527d8 [id=0fa5e1f97e984916a703c7b7da0f1087]) successfully deleted
I20250111 02:57:24.753098 19023 tablet_service.cc:1514] Processing DeleteTablet for tablet 94b9d14959df49a6bbd80ba9e9e57ee4 with delete_type TABLET_DATA_DELETED (replaced by tablet e4bd42ad66064bcaa043746081f1c759) from {username='slave'} at 127.0.0.1:36190
W20250111 02:57:24.746876 18629 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.66:37049 (request call id 36) took 988 ms (client timeout 1000 ms). Trace:
W20250111 02:57:24.756135 18629 rpcz_store.cc:269] 0111 02:57:23.758453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:23.758607 (+ 154us) service_pool.cc:225] Handling call
0111 02:57:24.746795 (+988188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2704,"reported_tablets":7}
I20250111 02:57:24.757880 18629 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_60c8bd1736a84a2d9fd52866d624704e" } modify_external_catalogs: true
I20250111 02:57:24.758498 18629 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_60c8bd1736a84a2d9fd52866d624704e" } modify_external_catalogs: true
I20250111 02:57:24.759992 19330 tablet_bootstrap.cc:654] T a983a670c2104398ac1a159ac3a1da7c P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:24.767623 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 131) took 3603 ms. Trace:
I20250111 02:57:24.767982 19326 rpcz_store.cc:276] 0111 02:57:21.164737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.164865 (+ 128us) service_pool.cc:225] Handling call
0111 02:57:24.691596 (+3526731us) ts_tablet_manager.cc:1145] Deleting tablet b54311f4269d4575b8998d7f545000e7
0111 02:57:24.691642 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:24.716562 (+ 24920us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.744606 (+ 28044us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.767555 (+ 22949us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3526170}
I20250111 02:57:24.769176 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet b54311f4269d4575b8998d7f545000e7 (table default.Table_292c103f235f421db8dacf834cca2c53 [id=b091f6cd63104693abbdd739bca159c3]) successfully deleted
I20250111 02:57:24.769910 19326 tablet_replica.cc:331] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:24.774005 19326 raft_consensus.cc:2238] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:24.774554 19321 tablet_bootstrap.cc:492] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:24.774924 19326 raft_consensus.cc:2267] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:24.775022 19321 ts_tablet_manager.cc:1397] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.001s
I20250111 02:57:24.776839 19326 ts_tablet_manager.cc:1905] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:24.778200 19321 raft_consensus.cc:357] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.779327 19321 raft_consensus.cc:383] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.779807 19321 raft_consensus.cc:738] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:24.780544 19321 consensus_queue.cc:260] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:24.662341 18909 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.007s sys 0.005s
W20250111 02:57:24.781688 18909 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s user 0.007s sys 0.005s
I20250111 02:57:24.783807 19321 ts_tablet_manager.cc:1428] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250111 02:57:24.783999 19695 consensus_queue.cc:1035] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:24.784840 19321 tablet_bootstrap.cc:492] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:24.785039 19330 tablet_bootstrap.cc:492] T a983a670c2104398ac1a159ac3a1da7c P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:24.785414 19330 ts_tablet_manager.cc:1397] T a983a670c2104398ac1a159ac3a1da7c P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.001s
I20250111 02:57:24.786953 19327 raft_consensus.cc:2238] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:24.788323 19330 raft_consensus.cc:357] T a983a670c2104398ac1a159ac3a1da7c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:24.789062 19330 raft_consensus.cc:383] T a983a670c2104398ac1a159ac3a1da7c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.789326 19330 raft_consensus.cc:738] T a983a670c2104398ac1a159ac3a1da7c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:24.792753 19321 tablet_bootstrap.cc:654] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:24.797137 19327 raft_consensus.cc:2267] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:24.798544 19330 consensus_queue.cc:260] T a983a670c2104398ac1a159ac3a1da7c P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:24.803990 19330 ts_tablet_manager.cc:1428] T a983a670c2104398ac1a159ac3a1da7c P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20250111 02:57:24.807775 19330 tablet_bootstrap.cc:492] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:24.813004 19327 ts_tablet_manager.cc:1905] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:24.816256 19321 tablet_bootstrap.cc:492] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:24.816684 19321 ts_tablet_manager.cc:1397] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.009s
I20250111 02:57:24.817248 19585 consensus_queue.cc:1035] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:24.819648 19321 raft_consensus.cc:357] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:24.820329 19321 raft_consensus.cc:383] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.822427 19321 raft_consensus.cc:738] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:24.823135 19321 consensus_queue.cc:260] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:24.832230 19321 ts_tablet_manager.cc:1428] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250111 02:57:24.833217 19321 tablet_bootstrap.cc:492] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:24.837913 19326 ts_tablet_manager.cc:1918] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:24.838281 19326 log.cc:1198] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/d8128b3941d54ce2a72bf2ae762a774f
I20250111 02:57:24.841470 19321 tablet_bootstrap.cc:654] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:24.843103 19330 tablet_bootstrap.cc:654] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:24.847492 18627 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 40fdec982cf643c68ef90798f4f65ad4 consensus state has no leader
I20250111 02:57:24.839006 19326 ts_tablet_manager.cc:1939] T d8128b3941d54ce2a72bf2ae762a774f P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:24.851266 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 132) took 3684 ms. Trace:
I20250111 02:57:24.851713 19326 rpcz_store.cc:276] 0111 02:57:21.166636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.166767 (+ 131us) service_pool.cc:225] Handling call
0111 02:57:24.769782 (+3603015us) ts_tablet_manager.cc:1145] Deleting tablet d8128b3941d54ce2a72bf2ae762a774f
0111 02:57:24.769813 (+ 31us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:24.811601 (+ 41788us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.837681 (+ 26080us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.851181 (+ 13500us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3602421}
I20250111 02:57:24.855674 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet d8128b3941d54ce2a72bf2ae762a774f (table default.Table_8b6452ba06af45bab320b50842e527d8 [id=0fa5e1f97e984916a703c7b7da0f1087]) successfully deleted
W20250111 02:57:24.859651 18622 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.67:53171 (request call id 37) took 1045 ms (client timeout 1000 ms). Trace:
W20250111 02:57:24.860102 18622 rpcz_store.cc:269] 0111 02:57:23.814469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:23.814628 (+ 159us) service_pool.cc:225] Handling call
0111 02:57:24.859580 (+1044952us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:24.497408 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:24.677916 (+180508us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:24.677941 (+ 25us) write_op.cc:435] Acquired schema lock
0111 02:57:24.677955 (+ 14us) tablet.cc:659] Decoding operations
0111 02:57:24.678244 (+ 289us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:24.678294 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:24.678316 (+ 22us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:24.678455 (+ 139us) tablet.cc:698] Row locks acquired
0111 02:57:24.678468 (+ 13us) write_op.cc:260] PREPARE: finished
0111 02:57:24.678588 (+ 120us) write_op.cc:270] Start()
0111 02:57:24.678672 (+ 84us) write_op.cc:276] Timestamp: P: 1736564244678548 usec, L: 0
0111 02:57:24.678688 (+ 16us) op_driver.cc:348] REPLICATION: starting
0111 02:57:24.679281 (+ 593us) log.cc:844] Serialized 356 byte log entry
0111 02:57:24.740070 (+ 60789us) op_driver.cc:464] REPLICATION: finished
0111 02:57:24.740320 (+ 250us) write_op.cc:301] APPLY: starting
0111 02:57:24.740381 (+ 61us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:24.740503 (+ 122us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:24.740613 (+ 110us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:24.740812 (+ 199us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:24.740956 (+ 144us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:24.741005 (+ 49us) write_op.cc:312] APPLY: finished
0111 02:57:24.741199 (+ 194us) log.cc:844] Serialized 23 byte log entry
0111 02:57:24.741340 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:24.741409 (+ 69us) write_op.cc:454] Released schema lock
0111 02:57:24.741458 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:24.747077 (+ 5619us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:24.747113 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":95,"apply.run_cpu_time_us":1377,"apply.run_wall_time_us":1375,"mutex_wait_us":12633,"num_ops":1,"prepare.queue_time_us":192281,"prepare.run_cpu_time_us":2348,"prepare.run_wall_time_us":249617,"raft.queue_time_us":12901,"raft.run_cpu_time_us":674,"raft.run_wall_time_us":676,"replication_time_us":61292}]]}
I20250111 02:57:24.863659 19326 tablet_replica.cc:331] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:24.864502 19167 tablet_service.cc:1514] Processing DeleteTablet for tablet c28958a1018344a48d18e6c0f03cd0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:17 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:24.864748 19326 raft_consensus.cc:2238] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:24.866154 19167 tablet_service.cc:1514] Processing DeleteTablet for tablet 94b9d14959df49a6bbd80ba9e9e57ee4 with delete_type TABLET_DATA_DELETED (replaced by tablet e4bd42ad66064bcaa043746081f1c759) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:24.866719 19326 raft_consensus.cc:2267] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:24.873939 19330 tablet_bootstrap.cc:492] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:24.874666 19330 ts_tablet_manager.cc:1397] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.067s user 0.012s sys 0.004s
I20250111 02:57:24.869803 19326 ts_tablet_manager.cc:1905] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:24.888245 19585 consensus_queue.cc:1035] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:24.889700 19321 tablet_bootstrap.cc:492] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:24.890137 19321 ts_tablet_manager.cc:1397] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.057s user 0.022s sys 0.030s
I20250111 02:57:24.892997 19321 raft_consensus.cc:357] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.893884 19321 raft_consensus.cc:383] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.894212 19321 raft_consensus.cc:738] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:24.891829 19585 raft_consensus.cc:491] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:24.895037 19321 consensus_queue.cc:260] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.878517 19330 raft_consensus.cc:357] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.896178 19330 raft_consensus.cc:383] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.896493 19330 raft_consensus.cc:738] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:24.899399 19321 ts_tablet_manager.cc:1428] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.009s user 0.002s sys 0.008s
I20250111 02:57:24.899443 19053 raft_consensus.cc:1270] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 2 FOLLOWER]: Refusing update from remote peer 8221604733fe4fc0a1fc32de8d6af71c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250111 02:57:24.900339 19321 tablet_bootstrap.cc:492] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:24.902690 19706 consensus_queue.cc:1035] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:24.903767 19330 consensus_queue.cc:260] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.911154 19321 tablet_bootstrap.cc:654] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:24.911094 19707 consensus_queue.cc:1035] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:24.914827 19330 ts_tablet_manager.cc:1428] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.040s user 0.007s sys 0.000s
I20250111 02:57:24.921583 19330 tablet_bootstrap.cc:492] T 9d6de202f4e547a0b8b90cfa09ad319a P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:24.928907 19330 tablet_bootstrap.cc:654] T 9d6de202f4e547a0b8b90cfa09ad319a P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:24.896227 19585 raft_consensus.cc:513] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:24.941076 18989 consensus_peers.cc:487] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fe4e60db4244a9a9696153b630c575b. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:24.944193 19326 ts_tablet_manager.cc:1918] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250111 02:57:24.944633 19326 log.cc:1198] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/5baef8688ede42fca28cfb6b0bdeefa6
I20250111 02:57:24.945376 19326 ts_tablet_manager.cc:1939] T 5baef8688ede42fca28cfb6b0bdeefa6 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:24.946127 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 128) took 3786 ms. Trace:
W20250111 02:57:24.946766 18989 consensus_peers.cc:487] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:24.948386 19321 tablet_bootstrap.cc:492] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:24.948909 19321 ts_tablet_manager.cc:1397] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.012s
I20250111 02:57:24.949246 19322 tablet_bootstrap.cc:492] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:24.949661 19322 ts_tablet_manager.cc:1397] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.813s user 0.012s sys 0.005s
W20250111 02:57:24.951131 18841 consensus_peers.cc:487] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef70b417568049f0b496f4d8e5f7e0e4. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:24.952569 19322 raft_consensus.cc:357] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.952556 19321 raft_consensus.cc:357] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:24.953279 19322 raft_consensus.cc:383] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.953537 19322 raft_consensus.cc:738] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:24.953565 19321 raft_consensus.cc:383] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:24.953912 19321 raft_consensus.cc:738] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:24.954195 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9192160e931445f1acd207604192f759" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:24.955530 19051 raft_consensus.cc:2463] T 9192160e931445f1acd207604192f759 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:24.955363 19321 consensus_queue.cc:260] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:24.957414 19131 leader_election.cc:304] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:24.958425 19321 ts_tablet_manager.cc:1428] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20250111 02:57:24.958868 19695 raft_consensus.cc:2798] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:24.959326 19695 raft_consensus.cc:491] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:24.959731 19695 raft_consensus.cc:3054] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:24.968283 19321 tablet_bootstrap.cc:492] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:24.970667 19696 raft_consensus.cc:491] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:24.946419 19326 rpcz_store.cc:276] 0111 02:57:21.160065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.176522 (+ 16457us) service_pool.cc:225] Handling call
0111 02:57:24.863426 (+3686904us) ts_tablet_manager.cc:1145] Deleting tablet 5baef8688ede42fca28cfb6b0bdeefa6
0111 02:57:24.863488 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:24.927790 (+ 64302us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.943966 (+ 16176us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:24.946073 (+ 2107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3685695}
I20250111 02:57:24.971660 19696 raft_consensus.cc:513] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.972256 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 5baef8688ede42fca28cfb6b0bdeefa6 (table default.Table_7f4578da9f5e418c9d67a535471b45cd [id=154c37dc0dba44bdab5e0315756dc7cf]) successfully deleted
I20250111 02:57:24.975849 19695 raft_consensus.cc:513] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:24.979673 19326 tablet_replica.cc:331] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:24.980537 19326 raft_consensus.cc:2238] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:24.986766 18905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9192160e931445f1acd207604192f759" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:24.991560 19133 leader_election.cc:343] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:24.990919 19321 tablet_bootstrap.cc:654] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:25.004244 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9192160e931445f1acd207604192f759" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:25.004994 19051 raft_consensus.cc:3054] T 9192160e931445f1acd207604192f759 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:25.014199 19051 raft_consensus.cc:2463] T 9192160e931445f1acd207604192f759 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:25.015479 19322 consensus_queue.cc:260] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:25.016072 19131 leader_election.cc:304] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:25.018033 19322 ts_tablet_manager.cc:1428] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.068s user 0.006s sys 0.001s
W20250111 02:57:25.027088 18841 consensus_peers.cc:487] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:25.023085 19585 leader_election.cc:290] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:25.034965 19326 raft_consensus.cc:2267] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:25.037181 19326 ts_tablet_manager.cc:1905] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:24.998477 19752 raft_consensus.cc:491] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:25.055100 19585 raft_consensus.cc:491] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:25.065289 18905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9192160e931445f1acd207604192f759" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:25.067358 19133 leader_election.cc:343] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:25.071877 18905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe8aa967fcb4bd7a48f6f0f69bea496" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:25.073563 19133 leader_election.cc:343] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:25.076695 19321 tablet_bootstrap.cc:492] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:25.077104 19321 ts_tablet_manager.cc:1397] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.109s user 0.011s sys 0.011s
I20250111 02:57:25.075672 19754 raft_consensus.cc:2798] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:25.078140 19327 ts_tablet_manager.cc:1918] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:25.068657 19695 leader_election.cc:290] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:25.078606 19327 log.cc:1198] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/29561e984e6d40fea032075f0531dd1f
I20250111 02:57:25.079417 19327 ts_tablet_manager.cc:1939] T 29561e984e6d40fea032075f0531dd1f P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:25.080184 19322 tablet_bootstrap.cc:492] T 40fdec982cf643c68ef90798f4f65ad4 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:25.096282 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 125) took 3905 ms. Trace:
I20250111 02:57:25.098991 19327 rpcz_store.cc:276] 0111 02:57:21.190848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.191122 (+ 274us) service_pool.cc:225] Handling call
0111 02:57:24.751888 (+3560766us) ts_tablet_manager.cc:1145] Deleting tablet 29561e984e6d40fea032075f0531dd1f
0111 02:57:24.751932 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:24.796871 (+ 44939us) spinlock_profiling.cc:243] Waited 34.1 ms on lock 0x7b54001806d4. stack: 00007f02faaf7853 00007f02faaf7750 00007f02fa66c584 00000000004fcd24 00000000004fccb9 00007f03019f76d1 00007f0301a7a352 00007f0301e971ac 00007f0302aa7d5a 00007f0302ad7ca0 00007f0302abee6d 00007f0302abee19 00007f0302abeda9 00007f0302abed71 00007f0302abdf9d 000000000059ea04
0111 02:57:24.820988 (+ 24117us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:25.077888 (+256900us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:25.096212 (+ 18324us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78328576,"tablet-delete.queue_time_us":3559893}
I20250111 02:57:25.103291 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 29561e984e6d40fea032075f0531dd1f (table default.Table_acc960ebdd02430ab5aa41a4f75f8578 [id=dc8029d5413f4e838d330383cf0b6f70]) successfully deleted
I20250111 02:57:25.104934 19677 raft_consensus.cc:491] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:25.105423 19677 raft_consensus.cc:513] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:25.063854 19585 raft_consensus.cc:513] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:25.074940 19696 leader_election.cc:290] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:25.113341 19585 leader_election.cc:290] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:25.056077 19752 raft_consensus.cc:513] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:25.116722 19677 leader_election.cc:290] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:25.123975 19752 leader_election.cc:290] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:25.124163 19321 raft_consensus.cc:357] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.124922 19321 raft_consensus.cc:383] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:25.125193 19321 raft_consensus.cc:738] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:25.125787 19321 consensus_queue.cc:260] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.126124 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199ca8692f39445f93d8c27e2134a32c" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:25.129446 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a983a670c2104398ac1a159ac3a1da7c" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:25.130179 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74592f80afab4513a9eb2eac351c4de5" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:25.130616 19327 tablet_replica.cc:331] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:25.131439 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199ca8692f39445f93d8c27e2134a32c" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:25.126844 19197 raft_consensus.cc:2463] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:25.132903 19754 raft_consensus.cc:695] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:25.136059 18986 leader_election.cc:304] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:25.136837 19677 raft_consensus.cc:2798] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250111 02:57:25.135022 19133 consensus_peers.cc:487] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:25.137147 19677 raft_consensus.cc:491] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:25.137231 19321 ts_tablet_manager.cc:1428] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.060s user 0.005s sys 0.002s
I20250111 02:57:25.137428 19677 raft_consensus.cc:3054] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:25.137444 19133 leader_election.cc:343] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:25.138093 19133 leader_election.cc:343] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:25.138238 19321 tablet_bootstrap.cc:492] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:25.142202 19326 ts_tablet_manager.cc:1918] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250111 02:57:25.131754 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 115819us
I20250111 02:57:25.133936 19754 consensus_queue.cc:237] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:25.152351 19321 tablet_bootstrap.cc:654] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:25.148164 18989 leader_election.cc:343] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:25.150743 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74592f80afab4513a9eb2eac351c4de5" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:25.151654 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe8aa967fcb4bd7a48f6f0f69bea496" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:25.152354 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a983a670c2104398ac1a159ac3a1da7c" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:25.156198 19050 raft_consensus.cc:2463] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:25.166558 19051 raft_consensus.cc:2463] T a983a670c2104398ac1a159ac3a1da7c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:25.173134 19131 leader_election.cc:304] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:25.173808 19325 ts_tablet_manager.cc:1918] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:25.174160 19325 log.cc:1198] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/b8e8c72c99ac4c369957b571d74baffe
I20250111 02:57:25.174839 19325 ts_tablet_manager.cc:1939] T b8e8c72c99ac4c369957b571d74baffe P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:25.174352 19585 raft_consensus.cc:2798] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:25.176541 19585 raft_consensus.cc:491] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:25.176990 19585 raft_consensus.cc:3054] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:25.179786 19322 tablet_bootstrap.cc:654] T 40fdec982cf643c68ef90798f4f65ad4 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:25.191779 19131 leader_election.cc:304] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:25.192696 19752 raft_consensus.cc:2798] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:25.193037 19752 raft_consensus.cc:491] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:25.193361 19752 raft_consensus.cc:3054] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:25.193751 19330 tablet_bootstrap.cc:492] T 9d6de202f4e547a0b8b90cfa09ad319a P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:25.159349 19054 raft_consensus.cc:2463] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:25.194226 19330 ts_tablet_manager.cc:1397] T 9d6de202f4e547a0b8b90cfa09ad319a P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.273s user 0.031s sys 0.045s
I20250111 02:57:25.153295 19326 log.cc:1198] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/29561e984e6d40fea032075f0531dd1f
I20250111 02:57:25.195652 19326 ts_tablet_manager.cc:1939] T 29561e984e6d40fea032075f0531dd1f P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:25.158767 19677 raft_consensus.cc:513] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:25.205615 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199ca8692f39445f93d8c27e2134a32c" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:25.207410 18989 leader_election.cc:343] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:25.211815 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet b8e8c72c99ac4c369957b571d74baffe (table default.Table_35d7f864912b40bcb5251a90d40c7849 [id=772d994bee2d4e34bd48912acecf8d48]) successfully deleted
I20250111 02:57:25.212049 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_dfe42fac0a764052a8d2f6add8882b64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:25.213791 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfe42fac0a764052a8d2f6add8882b64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:25.213760 19131 leader_election.cc:304] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
W20250111 02:57:25.216166 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet c28958a1018344a48d18e6c0f03cd0a3 because the tablet was not found. No further retry: Not found: Tablet not found: c28958a1018344a48d18e6c0f03cd0a3
I20250111 02:57:25.216373 19325 tablet_replica.cc:331] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:25.217075 19754 raft_consensus.cc:2798] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:25.217164 18630 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_42756e535547448f9335dab7bd96f772" } new_table_name: "default.Table_bccba04b48cc408d845624ee7ed98e5f" modify_external_catalogs: true
I20250111 02:57:25.217626 19754 raft_consensus.cc:491] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:25.218063 19754 raft_consensus.cc:3054] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:25.217767 19585 raft_consensus.cc:513] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:25.217144 19325 raft_consensus.cc:2238] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:25.219538 19325 raft_consensus.cc:2267] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:25.205489 19752 raft_consensus.cc:513] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:25.199585 19330 raft_consensus.cc:357] T 9d6de202f4e547a0b8b90cfa09ad319a P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.221283 19325 ts_tablet_manager.cc:1905] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:25.221524 19330 raft_consensus.cc:383] T 9d6de202f4e547a0b8b90cfa09ad319a P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:25.221832 19330 raft_consensus.cc:738] T 9d6de202f4e547a0b8b90cfa09ad319a P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:25.222820 19752 leader_election.cc:290] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:25.222610 19330 consensus_queue.cc:260] T 9d6de202f4e547a0b8b90cfa09ad319a P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.223979 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a983a670c2104398ac1a159ac3a1da7c" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:25.224419 18625 catalog_manager.cc:5526] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0cedab82b154d62b78809edc6f28bfa (127.18.20.67). New cstate: current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } }
I20250111 02:57:25.227349 18625 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]). Expected version 2 got 0
I20250111 02:57:25.227977 19322 tablet_bootstrap.cc:492] T 40fdec982cf643c68ef90798f4f65ad4 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:25.229828 19322 ts_tablet_manager.cc:1397] T 40fdec982cf643c68ef90798f4f65ad4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.150s user 0.009s sys 0.002s
I20250111 02:57:25.232076 19327 raft_consensus.cc:2238] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:25.233090 19327 raft_consensus.cc:2267] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:25.232864 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a983a670c2104398ac1a159ac3a1da7c" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:25.233793 19054 raft_consensus.cc:3054] T a983a670c2104398ac1a159ac3a1da7c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:25.225379 19133 leader_election.cc:343] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:25.226739 19321 tablet_bootstrap.cc:492] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:25.234560 19321 ts_tablet_manager.cc:1397] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.096s user 0.012s sys 0.004s
I20250111 02:57:25.240828 19321 raft_consensus.cc:357] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.241585 19321 raft_consensus.cc:383] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:25.241848 19321 raft_consensus.cc:738] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:25.242110 19322 raft_consensus.cc:357] T 40fdec982cf643c68ef90798f4f65ad4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:25.244102 19330 ts_tablet_manager.cc:1428] T 9d6de202f4e547a0b8b90cfa09ad319a P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.050s user 0.005s sys 0.003s
I20250111 02:57:25.244180 19322 raft_consensus.cc:383] T 40fdec982cf643c68ef90798f4f65ad4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:25.244480 19322 raft_consensus.cc:738] T 40fdec982cf643c68ef90798f4f65ad4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:25.245040 19330 tablet_bootstrap.cc:492] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:25.242419 19321 consensus_queue.cc:260] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.248884 19327 ts_tablet_manager.cc:1905] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:25.251643 19321 ts_tablet_manager.cc:1428] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20250111 02:57:25.252341 19322 consensus_queue.cc:260] T 40fdec982cf643c68ef90798f4f65ad4 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:25.254832 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74592f80afab4513a9eb2eac351c4de5" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:25.257068 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74592f80afab4513a9eb2eac351c4de5" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:25.257879 19051 raft_consensus.cc:3054] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:25.263239 18623 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 74592f80afab4513a9eb2eac351c4de5
I20250111 02:57:25.264690 18623 catalog_manager.cc:6596] ReplaceTablet: tablet 74592f80afab4513a9eb2eac351c4de5 deleted and replaced by tablet f57996ef57164187b7afbff68c1b934c
I20250111 02:57:25.265544 18623 master_service.cc:946] ReplaceTablet: received request to replace tablet 199ca8692f39445f93d8c27e2134a32c from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:25.266109 19023 tablet_service.cc:1514] Processing DeleteTablet for tablet 74592f80afab4513a9eb2eac351c4de5 with delete_type TABLET_DATA_DELETED (replaced by tablet f57996ef57164187b7afbff68c1b934c) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:25.267352 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 74592f80afab4513a9eb2eac351c4de5 with delete_type TABLET_DATA_DELETED (replaced by tablet f57996ef57164187b7afbff68c1b934c) from {username='slave'} at 127.0.0.1:51082
W20250111 02:57:25.256712 19133 leader_election.cc:343] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:25.267364 19585 leader_election.cc:290] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:25.268556 19177 tablet_service.cc:1514] Processing DeleteTablet for tablet 74592f80afab4513a9eb2eac351c4de5 with delete_type TABLET_DATA_DELETED (replaced by tablet f57996ef57164187b7afbff68c1b934c) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:25.272475 19321 tablet_bootstrap.cc:492] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:25.276681 19754 raft_consensus.cc:513] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:25.280380 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe8aa967fcb4bd7a48f6f0f69bea496" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:25.281143 19050 raft_consensus.cc:3054] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:25.290948 19677 leader_election.cc:290] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:25.291154 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199ca8692f39445f93d8c27e2134a32c" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:25.291998 19197 raft_consensus.cc:3054] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:25.295820 19322 ts_tablet_manager.cc:1428] T 40fdec982cf643c68ef90798f4f65ad4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.065s user 0.004s sys 0.002s
I20250111 02:57:25.298417 19050 raft_consensus.cc:2463] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:25.301453 19322 tablet_bootstrap.cc:492] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:25.304163 19131 leader_election.cc:304] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:25.306059 19752 raft_consensus.cc:2798] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:25.306681 19752 raft_consensus.cc:695] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:25.311146 19051 raft_consensus.cc:2463] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:25.307765 19752 consensus_queue.cc:237] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:25.317062 19754 leader_election.cc:290] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:25.317867 19131 leader_election.cc:304] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:25.318406 19696 consensus_queue.cc:1035] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:25.319969 19696 raft_consensus.cc:2798] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:25.319805 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe8aa967fcb4bd7a48f6f0f69bea496" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:25.320485 19696 raft_consensus.cc:695] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:25.197597 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 133) took 4000 ms. Trace:
I20250111 02:57:25.323814 19326 rpcz_store.cc:276] 0111 02:57:21.196884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.197024 (+ 140us) service_pool.cc:225] Handling call
0111 02:57:24.979477 (+3782453us) ts_tablet_manager.cc:1145] Deleting tablet 29561e984e6d40fea032075f0531dd1f
0111 02:57:24.979543 (+ 66us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:25.086007 (+106464us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:25.141945 (+ 55938us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:25.197536 (+ 55591us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3781752}
I20250111 02:57:25.321492 19696 consensus_queue.cc:237] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:25.328656 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 29561e984e6d40fea032075f0531dd1f (table default.Table_acc960ebdd02430ab5aa41a4f75f8578 [id=dc8029d5413f4e838d330383cf0b6f70]) successfully deleted
W20250111 02:57:25.328475 19133 leader_election.cc:343] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:25.335291 19327 ts_tablet_manager.cc:1918] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:25.337201 19327 log.cc:1198] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/b54311f4269d4575b8998d7f545000e7
I20250111 02:57:25.338198 19327 ts_tablet_manager.cc:1939] T b54311f4269d4575b8998d7f545000e7 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:25.340691 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 126) took 4098 ms. Trace:
I20250111 02:57:25.336874 19197 raft_consensus.cc:2463] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:25.341056 19327 rpcz_store.cc:276] 0111 02:57:21.241983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.242141 (+ 158us) service_pool.cc:225] Handling call
0111 02:57:25.119797 (+3877656us) ts_tablet_manager.cc:1145] Deleting tablet b54311f4269d4575b8998d7f545000e7
0111 02:57:25.119844 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:25.310638 (+190794us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:25.335045 (+ 24407us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:25.340620 (+ 5575us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3877018}
I20250111 02:57:25.342900 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet b54311f4269d4575b8998d7f545000e7 (table default.Table_292c103f235f421db8dacf834cca2c53 [id=b091f6cd63104693abbdd739bca159c3]) successfully deleted
I20250111 02:57:25.342893 18626 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_ea068446b5434b3ca5f0afa8386c0655"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:25.345192 18626 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea068446b5434b3ca5f0afa8386c0655 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:25.348374 19322 tablet_bootstrap.cc:654] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:25.349262 19327 tablet_replica.cc:331] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:25.350144 19327 raft_consensus.cc:2238] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:25.350318 19776 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.20.126:45115 every 8 connection(s)
I20250111 02:57:25.350816 19327 raft_consensus.cc:2267] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:25.355008 19327 ts_tablet_manager.cc:1905] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:25.358934 18624 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3514d795a0f24adcaa2accc0cf05b142 has no consensus state
I20250111 02:57:25.362227 19309 rpc_server.cc:307] RPC server started. Bound to: 127.18.20.126:45115
I20250111 02:57:25.363472 18986 leader_election.cc:304] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:25.365012 19677 raft_consensus.cc:2798] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:25.367231 19677 raft_consensus.cc:695] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:25.362169 19585 raft_consensus.cc:491] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:25.368314 19585 raft_consensus.cc:513] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.371110 19321 tablet_bootstrap.cc:654] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:25.371704 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6de202f4e547a0b8b90cfa09ad319a" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:25.372201 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6de202f4e547a0b8b90cfa09ad319a" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:25.373042 19051 raft_consensus.cc:2463] T 9d6de202f4e547a0b8b90cfa09ad319a P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
W20250111 02:57:25.373422 19133 leader_election.cc:343] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:25.374807 19326 tablet_replica.cc:331] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:25.374632 19131 leader_election.cc:304] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:25.370836 19585 leader_election.cc:290] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:25.375619 19585 raft_consensus.cc:2798] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:25.375718 19326 raft_consensus.cc:2238] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:25.376017 19585 raft_consensus.cc:491] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:25.376400 19585 raft_consensus.cc:3054] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:25.376807 19326 raft_consensus.cc:2267] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:25.378753 19326 ts_tablet_manager.cc:1905] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:25.383173 19327 ts_tablet_manager.cc:1918] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:25.383579 19327 log.cc:1198] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/c28958a1018344a48d18e6c0f03cd0a3
I20250111 02:57:25.384334 19327 ts_tablet_manager.cc:1939] T c28958a1018344a48d18e6c0f03cd0a3 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:25.386291 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 132) took 3934 ms. Trace:
I20250111 02:57:25.386600 19327 rpcz_store.cc:276] 0111 02:57:21.451841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.452013 (+ 172us) service_pool.cc:225] Handling call
0111 02:57:25.349093 (+3897080us) ts_tablet_manager.cc:1145] Deleting tablet c28958a1018344a48d18e6c0f03cd0a3
0111 02:57:25.349141 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:25.366186 (+ 17045us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:25.382873 (+ 16687us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:25.386230 (+ 3357us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3896312}
I20250111 02:57:25.390184 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 139) took 3705 ms. Trace:
I20250111 02:57:25.390578 19327 rpcz_store.cc:276] 0111 02:57:21.684388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.684508 (+ 120us) service_pool.cc:225] Handling call
0111 02:57:25.389798 (+3705290us) ts_tablet_manager.cc:1145] Deleting tablet 2e06e867266948df8eb35a85c58f578f
0111 02:57:25.389842 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:25.390121 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3704562}
I20250111 02:57:25.391788 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 140) took 3444 ms. Trace:
I20250111 02:57:25.392109 19327 rpcz_store.cc:276] 0111 02:57:21.948033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.971198 (+ 23165us) service_pool.cc:225] Handling call
0111 02:57:25.391460 (+3420262us) ts_tablet_manager.cc:1145] Deleting tablet 4b302fcba27244c193586bc25fc9305f
0111 02:57:25.391497 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:25.391738 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3419082}
I20250111 02:57:25.393000 19327 tablet_replica.cc:331] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:25.393368 19321 tablet_bootstrap.cc:492] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:25.393782 19321 ts_tablet_manager.cc:1397] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.122s user 0.008s sys 0.004s
I20250111 02:57:25.393779 19327 raft_consensus.cc:2238] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:25.394608 19327 raft_consensus.cc:2267] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:25.396194 19321 raft_consensus.cc:357] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.396855 19321 raft_consensus.cc:383] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:25.397110 19321 raft_consensus.cc:738] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:25.397012 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_0dcad07d56d843e59f351d933e625a21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:25.397823 19309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/master-0/data/info.pb
I20250111 02:57:25.397765 19321 consensus_queue.cc:260] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:25.398514 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dcad07d56d843e59f351d933e625a21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:25.399791 19321 ts_tablet_manager.cc:1428] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250111 02:57:25.400605 19321 tablet_bootstrap.cc:492] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:25.403102 19327 ts_tablet_manager.cc:1905] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:25.403497 18513 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu as pid 19309
I20250111 02:57:25.437132 19325 ts_tablet_manager.cc:1918] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:25.437582 19325 log.cc:1198] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/94b9d14959df49a6bbd80ba9e9e57ee4
I20250111 02:57:25.438298 19325 ts_tablet_manager.cc:1939] T 94b9d14959df49a6bbd80ba9e9e57ee4 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:25.438768 19696 raft_consensus.cc:491] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:25.439245 19696 raft_consensus.cc:513] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.381345 19677 consensus_queue.cc:237] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:25.388569 18986 consensus_peers.cc:487] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c28958a1018344a48d18e6c0f03cd0a3. This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:25.440030 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 51670.8us
I20250111 02:57:25.440857 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 94b9d14959df49a6bbd80ba9e9e57ee4 (table default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7]) successfully deleted
I20250111 02:57:25.445199 19696 leader_election.cc:290] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:25.445780 19752 consensus_queue.cc:1035] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:25.446096 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7158a7fc624fa4bc816ac3721c610d" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:25.449736 19131 leader_election.cc:343] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:25.451277 19585 raft_consensus.cc:513] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.391464 19330 tablet_bootstrap.cc:654] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:25.454085 19752 consensus_queue.cc:1035] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:25.455511 19325 tablet_replica.cc:331] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:25.394816 19054 raft_consensus.cc:2463] T a983a670c2104398ac1a159ac3a1da7c P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:25.456348 19325 raft_consensus.cc:2238] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:25.457072 19325 pending_rounds.cc:62] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa: Trying to abort 1 pending ops.
I20250111 02:57:25.457298 19321 tablet_bootstrap.cc:654] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:25.457319 19325 pending_rounds.cc:69] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112967148875870208 op_type: NO_OP noop_request { }
I20250111 02:57:25.469686 19325 raft_consensus.cc:2883] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:57:25.470193 19325 raft_consensus.cc:2267] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:25.470697 19585 leader_election.cc:290] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:25.474401 19321 tablet_bootstrap.cc:492] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:25.474825 19321 ts_tablet_manager.cc:1397] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.000s
I20250111 02:57:25.477156 19325 ts_tablet_manager.cc:1905] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:25.488154 19321 raft_consensus.cc:357] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.490078 19321 raft_consensus.cc:383] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:25.490391 19321 raft_consensus.cc:738] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:25.491202 19321 consensus_queue.cc:260] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.494252 19585 raft_consensus.cc:491] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:25.494936 19585 raft_consensus.cc:513] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.498484 19585 leader_election.cc:290] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:25.501358 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet c28958a1018344a48d18e6c0f03cd0a3 (table default.Table_7bf5032295a34c379d00c2d9ad7e417f [id=f6f9076d18654f339a6feac0c468e265]) successfully deleted
W20250111 02:57:25.502067 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 2e06e867266948df8eb35a85c58f578f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e06e867266948df8eb35a85c58f578f already in progress: creating tablet
I20250111 02:57:25.503654 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_b19a186d285d4b5b845727344f7d155b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:25.505316 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b19a186d285d4b5b845727344f7d155b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:25.513020 19321 ts_tablet_manager.cc:1428] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20250111 02:57:25.513947 19321 tablet_bootstrap.cc:492] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:25.523979 18884 tablet_service.cc:1514] Processing DeleteTablet for tablet b54311f4269d4575b8998d7f545000e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 845da3ee0863430ea3fe9de8c84f2c17) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:25.530296 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 29561e984e6d40fea032075f0531dd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:15 UTC) from {username='slave'} at 127.0.0.1:51082
W20250111 02:57:25.532819 19133 consensus_peers.cc:487] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:25.534516 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a4bc8e9c0c947f6861a01d3e3bd7f31" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:25.537199 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7158a7fc624fa4bc816ac3721c610d" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:25.538827 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6de202f4e547a0b8b90cfa09ad319a" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:25.545235 19133 leader_election.cc:343] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:25.545928 19133 leader_election.cc:343] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:25.546301 19133 leader_election.cc:304] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:25.548156 19585 raft_consensus.cc:2743] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:25.555595 19133 leader_election.cc:343] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:25.563482 19757 raft_consensus.cc:491] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:25.563999 19757 raft_consensus.cc:513] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.565249 18626 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_fcfe588eae964e7b9f5af53b5728e639" } new_table_name: "default.Table_49e4dd58d29f4cb6a0ed4f85138713b8" modify_external_catalogs: true
I20250111 02:57:25.566051 19757 leader_election.cc:290] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:25.568048 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32afc2f158254811836fcf00f7f409ed" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:25.570775 19133 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:25.570977 18960 heartbeater.cc:346] Connected to a master server at 127.18.20.126:45115
I20250111 02:57:25.586213 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6de202f4e547a0b8b90cfa09ad319a" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:25.587451 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a4bc8e9c0c947f6861a01d3e3bd7f31" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:25.588534 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32afc2f158254811836fcf00f7f409ed" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:25.586864 19050 raft_consensus.cc:3054] T 9d6de202f4e547a0b8b90cfa09ad319a P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:25.563222 19325 ts_tablet_manager.cc:1918] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:25.590085 19325 log.cc:1198] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/74592f80afab4513a9eb2eac351c4de5
I20250111 02:57:25.590817 19325 ts_tablet_manager.cc:1939] T 74592f80afab4513a9eb2eac351c4de5 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:25.591518 19131 leader_election.cc:304] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:25.592067 19602 raft_consensus.cc:491] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:25.592599 19602 raft_consensus.cc:513] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:25.594416 19131 leader_election.cc:343] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:25.597952 18842 leader_election.cc:343] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: 92aa64b2de5746d1b12cc93a10c9d56e
I20250111 02:57:25.594838 19131 leader_election.cc:304] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:25.599347 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 74592f80afab4513a9eb2eac351c4de5 (table default.Table_5bc6e85b9ba443aa8ddc113f6b5debbc [id=7c2fda5567344ad8b8034c9aabe10231]) successfully deleted
I20250111 02:57:25.599733 19696 raft_consensus.cc:2798] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:25.600176 19696 raft_consensus.cc:695] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:25.600967 19696 consensus_queue.cc:237] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:25.595009 19602 leader_election.cc:290] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:25.603261 19752 raft_consensus.cc:2743] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:25.604583 19131 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:25.604970 19131 leader_election.cc:304] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:25.605726 19752 raft_consensus.cc:2743] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:25.605885 19050 raft_consensus.cc:2463] T 9d6de202f4e547a0b8b90cfa09ad319a P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:25.607079 19322 tablet_bootstrap.cc:492] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:25.607597 19322 ts_tablet_manager.cc:1397] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.306s user 0.011s sys 0.009s
I20250111 02:57:25.610369 19244 heartbeater.cc:346] Connected to a master server at 127.18.20.126:45115
I20250111 02:57:25.610415 19322 raft_consensus.cc:357] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:25.610913 19244 heartbeater.cc:502] Master 127.18.20.126:45115 was elected leader, sending a full tablet report...
I20250111 02:57:25.611169 19322 raft_consensus.cc:383] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250111 02:57:25.611282 18614 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 3514d795a0f24adcaa2accc0cf05b142 (table default.Table_4c9aebb06d554107a5448d968fdb0436 [id=3fab00618e6548de85b5d20adee4e9ac], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3514d795a0f24adcaa2accc0cf05b142 has no consensus state
I20250111 02:57:25.611438 19322 raft_consensus.cc:738] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:25.619027 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aa64b2de5746d1b12cc93a10c9d56e" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:25.619753 19197 raft_consensus.cc:2463] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:25.621132 18841 leader_election.cc:304] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f
I20250111 02:57:25.622279 19602 raft_consensus.cc:2798] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:25.622609 19602 raft_consensus.cc:491] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:25.622900 19602 raft_consensus.cc:3054] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:25.628326 19322 consensus_queue.cc:260] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:25.629312 18624 master_service.cc:946] ReplaceTablet: received request to replace tablet 061e6eb5775443679e9cbd6b11f66fe3 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:25.641734 19131 leader_election.cc:304] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:25.644213 19322 ts_tablet_manager.cc:1428] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20250111 02:57:25.643327 19786 sys_catalog.cc:263] Verifying existing consensus state
I20250111 02:57:25.647090 19101 heartbeater.cc:346] Connected to a master server at 127.18.20.126:45115
I20250111 02:57:25.647377 19696 raft_consensus.cc:2798] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:25.648069 19696 raft_consensus.cc:695] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:25.648895 19696 consensus_queue.cc:237] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.650205 19322 tablet_bootstrap.cc:492] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:25.650316 19327 ts_tablet_manager.cc:1918] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:25.650707 19327 log.cc:1198] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/ef70b417568049f0b496f4d8e5f7e0e4
I20250111 02:57:25.651674 19327 ts_tablet_manager.cc:1939] T ef70b417568049f0b496f4d8e5f7e0e4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:25.652470 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 142) took 3603 ms. Trace:
I20250111 02:57:25.657963 19602 raft_consensus.cc:513] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:25.662098 19786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Bootstrap starting.
I20250111 02:57:25.689877 19321 tablet_bootstrap.cc:654] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:25.689824 19786 log.cc:826] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Log is configured to *not* fsync() on all Append() calls
I20250111 02:57:25.698830 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_535081a0b0ff46998f523371377b75f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:25.702685 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_535081a0b0ff46998f523371377b75f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:25.712592 18626 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 855dac47872d4c5fbf8f5e38b588c7fe consensus state has no leader
I20250111 02:57:25.728502 19786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 4 replicates
I20250111 02:57:25.729683 19786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Bootstrap complete.
I20250111 02:57:25.734125 19326 ts_tablet_manager.cc:1918] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:25.734623 19326 log.cc:1198] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/9fe4e60db4244a9a9696153b630c575b
I20250111 02:57:25.735474 19326 ts_tablet_manager.cc:1939] T 9fe4e60db4244a9a9696153b630c575b P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:25.652778 19327 rpcz_store.cc:276] 0111 02:57:22.048762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.048922 (+ 160us) service_pool.cc:225] Handling call
0111 02:57:25.392864 (+3343942us) ts_tablet_manager.cc:1145] Deleting tablet ef70b417568049f0b496f4d8e5f7e0e4
0111 02:57:25.392900 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:25.577633 (+184733us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:25.649993 (+ 72360us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:25.652395 (+ 2402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3343390}
I20250111 02:57:25.673573 19322 tablet_bootstrap.cc:654] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:25.746686 19327 tablet_replica.cc:331] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:25.747635 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet ef70b417568049f0b496f4d8e5f7e0e4 (table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5]) successfully deleted
I20250111 02:57:25.748647 19327 raft_consensus.cc:2238] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:25.749155 19327 raft_consensus.cc:2267] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:25.751019 19327 ts_tablet_manager.cc:1905] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:25.762665 18626 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_ee18021e6a544ed6b86b624aec8c39be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:25.767671 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aa64b2de5746d1b12cc93a10c9d56e" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:25.768309 19197 raft_consensus.cc:3054] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:25.776927 18842 leader_election.cc:343] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: 92aa64b2de5746d1b12cc93a10c9d56e
I20250111 02:57:25.779408 19197 raft_consensus.cc:2463] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:25.780668 18841 leader_election.cc:304] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f
I20250111 02:57:25.781507 19706 raft_consensus.cc:2798] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:25.739377 19330 tablet_bootstrap.cc:492] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:25.786909 19330 ts_tablet_manager.cc:1397] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.542s user 0.004s sys 0.017s
I20250111 02:57:25.789438 19330 raft_consensus.cc:357] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.790184 19330 raft_consensus.cc:383] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:25.790447 19330 raft_consensus.cc:738] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:25.786764 19786 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 1 FOLLOWER]: Replica starting. Triggering 4 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:25.794976 19706 raft_consensus.cc:695] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 LEADER]: Becoming Leader. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Running, Role: LEADER
I20250111 02:57:25.796000 19706 consensus_queue.cc:237] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:25.798888 18626 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee18021e6a544ed6b86b624aec8c39be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:25.807170 19602 leader_election.cc:290] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:25.814944 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_f9d19027fe814825a6c3906baef77ff3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:25.816627 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9d19027fe814825a6c3906baef77ff3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:25.819886 18989 consensus_peers.cc:487] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c28958a1018344a48d18e6c0f03cd0a3. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:25.826006 19786 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e336d4ddf044bf5b0b6e4f5af634727, State: Initialized, Role: FOLLOWER
I20250111 02:57:25.827646 19786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:25.851248 19786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:57:25.846185 19805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:25.854672 19805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:25.863232 19322 tablet_bootstrap.cc:492] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:25.863811 19322 ts_tablet_manager.cc:1397] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.214s user 0.025s sys 0.006s
I20250111 02:57:25.865964 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 134) took 4640 ms. Trace:
I20250111 02:57:25.880630 19330 consensus_queue.cc:260] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.886812 19322 raft_consensus.cc:357] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:25.889209 18909 raft_consensus.cc:3054] T 40fdec982cf643c68ef90798f4f65ad4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:25.887832 19322 raft_consensus.cc:383] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:25.892942 19322 raft_consensus.cc:738] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:25.903393 19322 consensus_queue.cc:260] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:25.866592 19326 rpcz_store.cc:276] 0111 02:57:21.225981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.226124 (+ 143us) service_pool.cc:225] Handling call
0111 02:57:25.374245 (+4148121us) ts_tablet_manager.cc:1145] Deleting tablet 9fe4e60db4244a9a9696153b630c575b
0111 02:57:25.374297 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:25.614003 (+239706us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:25.733903 (+119900us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:25.865867 (+131964us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4147582}
I20250111 02:57:25.906805 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 9fe4e60db4244a9a9696153b630c575b (table default.Table_064d4893c74f4870bc45572dd481c63c [id=b0a0ff2f4d554503a18d05e0ff51402d]) successfully deleted
I20250111 02:57:25.908546 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 146) took 4196 ms. Trace:
I20250111 02:57:25.909180 19326 rpcz_store.cc:276] 0111 02:57:21.711826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.718939 (+ 7113us) service_pool.cc:225] Handling call
0111 02:57:25.907878 (+4188939us) ts_tablet_manager.cc:1145] Deleting tablet 2e06e867266948df8eb35a85c58f578f
0111 02:57:25.907960 (+ 82us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:25.908450 (+ 490us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4188260}
I20250111 02:57:25.913232 19322 ts_tablet_manager.cc:1428] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.049s user 0.007s sys 0.000s
I20250111 02:57:25.914213 19322 tablet_bootstrap.cc:492] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:25.916914 18960 heartbeater.cc:346] Connected to a master server at 127.18.20.126:45115
I20250111 02:57:25.917520 19321 tablet_bootstrap.cc:492] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:25.917948 19321 ts_tablet_manager.cc:1397] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.404s user 0.007s sys 0.005s
I20250111 02:57:25.920454 19321 raft_consensus.cc:357] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.921151 19321 raft_consensus.cc:383] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:25.921559 19321 raft_consensus.cc:738] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:25.923552 19330 ts_tablet_manager.cc:1428] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.136s user 0.007s sys 0.000s
I20250111 02:57:25.928397 19330 tablet_bootstrap.cc:492] T 49616e4e7cc647509e72cde57b87d6fa P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:25.931211 19677 raft_consensus.cc:491] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:25.931800 19677 raft_consensus.cc:513] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.934293 19677 leader_election.cc:290] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:25.936383 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a4bc8e9c0c947f6861a01d3e3bd7f31" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:25.937280 19197 raft_consensus.cc:2463] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:25.938447 18986 leader_election.cc:304] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:25.939291 19677 raft_consensus.cc:2798] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:25.944444 19677 raft_consensus.cc:491] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:25.944972 19677 raft_consensus.cc:3054] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:25.963601 19244 heartbeater.cc:502] Master 127.18.20.126:45115 was elected leader, sending a full tablet report...
W20250111 02:57:25.963773 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 2e06e867266948df8eb35a85c58f578f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e06e867266948df8eb35a85c58f578f already in progress: creating tablet
I20250111 02:57:25.964704 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a4bc8e9c0c947f6861a01d3e3bd7f31" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:25.964952 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 147) took 4015 ms. Trace:
I20250111 02:57:25.965317 19326 rpcz_store.cc:276] 0111 02:57:21.949077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:21.949726 (+ 649us) service_pool.cc:225] Handling call
0111 02:57:25.964567 (+4014841us) ts_tablet_manager.cc:1145] Deleting tablet 4b302fcba27244c193586bc25fc9305f
0111 02:57:25.964617 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:25.964888 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4013959}
I20250111 02:57:25.967681 19322 tablet_bootstrap.cc:654] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:25.969477 18989 leader_election.cc:343] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:25.968919 19321 consensus_queue.cc:260] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:25.986826 19321 ts_tablet_manager.cc:1428] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.069s user 0.004s sys 0.003s
I20250111 02:57:25.987744 19330 tablet_bootstrap.cc:654] T 49616e4e7cc647509e72cde57b87d6fa P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:25.987845 19321 tablet_bootstrap.cc:492] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:25.991768 19327 ts_tablet_manager.cc:1918] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:25.992159 19327 log.cc:1198] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/0dbb63ab78f84995ae6d9fb2aae78dcf
I20250111 02:57:25.992825 19327 ts_tablet_manager.cc:1939] T 0dbb63ab78f84995ae6d9fb2aae78dcf P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:25.993618 19326 tablet_replica.cc:331] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:25.994421 19326 raft_consensus.cc:2238] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:25.994890 19326 raft_consensus.cc:2267] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:25.997285 19677 raft_consensus.cc:513] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.001884 19677 leader_election.cc:290] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:26.002945 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a4bc8e9c0c947f6861a01d3e3bd7f31" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:26.003592 19197 raft_consensus.cc:3054] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:26.002943 19683 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:57:26.014499 19197 raft_consensus.cc:2463] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:26.049372 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a4bc8e9c0c947f6861a01d3e3bd7f31" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:26.044356 19706 raft_consensus.cc:491] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:26.051680 19706 raft_consensus.cc:513] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:26.054477 19706 leader_election.cc:290] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:26.056602 19683 pending_rounds.cc:77] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Aborting all ops after (but not including) 5
I20250111 02:57:26.056975 19683 pending_rounds.cc:99] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Aborting uncommitted WRITE_OP operation due to leader change: 1.6
W20250111 02:57:26.057368 18841 leader_election.cc:343] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Not found: Tablet not found: 94b9d14959df49a6bbd80ba9e9e57ee4
I20250111 02:57:26.058451 19683 pending_rounds.cc:99] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Aborting uncommitted WRITE_OP operation due to leader change: 1.7
I20250111 02:57:26.059698 19683 pending_rounds.cc:99] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Aborting uncommitted WRITE_OP operation due to leader change: 1.8
I20250111 02:57:26.081238 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 141) took 4065 ms. Trace:
I20250111 02:57:26.081622 19327 rpcz_store.cc:276] 0111 02:57:22.015588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.057102 (+ 41514us) service_pool.cc:225] Handling call
0111 02:57:25.746468 (+3689366us) ts_tablet_manager.cc:1145] Deleting tablet 0dbb63ab78f84995ae6d9fb2aae78dcf
0111 02:57:25.746551 (+ 83us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:25.761346 (+ 14795us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:25.991519 (+230173us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:26.081170 (+ 89651us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3688492}
I20250111 02:57:26.082859 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 0dbb63ab78f84995ae6d9fb2aae78dcf (table default.Table_154c30276c8d4a8e805dc2fb85867bbf [id=9d3552a625804f0a9350454f3b5dc2e8]) successfully deleted
I20250111 02:57:26.084111 19327 tablet_replica.cc:331] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:26.084940 19327 raft_consensus.cc:2238] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:26.074767 19321 tablet_bootstrap.cc:654] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:26.087350 19327 raft_consensus.cc:2267] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:26.028466 19326 ts_tablet_manager.cc:1905] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:26.029863 18986 leader_election.cc:304] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
W20250111 02:57:26.089010 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 59238.1us
I20250111 02:57:26.056974 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b9d14959df49a6bbd80ba9e9e57ee4" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:26.090370 19054 raft_consensus.cc:2388] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
W20250111 02:57:26.091203 18989 leader_election.cc:343] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:26.092797 19481 raft_consensus.cc:2798] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:26.094493 18842 leader_election.cc:304] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:26.095862 19602 raft_consensus.cc:3054] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:26.101027 19322 tablet_bootstrap.cc:492] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:26.101667 19322 ts_tablet_manager.cc:1397] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.188s user 0.012s sys 0.036s
I20250111 02:57:26.104873 19327 ts_tablet_manager.cc:1905] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:26.106490 19322 raft_consensus.cc:357] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:26.107218 19322 raft_consensus.cc:383] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250111 02:57:26.107189 19133 consensus_peers.cc:487] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:26.107515 19322 raft_consensus.cc:738] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:26.113876 19322 consensus_queue.cc:260] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:26.116006 19322 ts_tablet_manager.cc:1428] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250111 02:57:26.119364 19602 raft_consensus.cc:2743] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:26.129928 19133 consensus_peers.cc:487] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:26.129911 19322 tablet_bootstrap.cc:492] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:26.145188 19327 ts_tablet_manager.cc:1918] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:26.145344 19321 tablet_bootstrap.cc:492] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:26.145606 19327 log.cc:1198] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/1b2377893bfc426fa00d5ddacd9c5961
I20250111 02:57:26.145820 19321 ts_tablet_manager.cc:1397] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.158s user 0.013s sys 0.005s
I20250111 02:57:26.146350 19327 ts_tablet_manager.cc:1939] T 1b2377893bfc426fa00d5ddacd9c5961 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:26.148878 19321 raft_consensus.cc:357] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:26.149641 19321 raft_consensus.cc:383] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:26.149919 19321 raft_consensus.cc:738] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:26.158890 19481 raft_consensus.cc:695] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:26.158428 19321 consensus_queue.cc:260] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:26.160292 19481 consensus_queue.cc:237] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.164490 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 143) took 3920 ms. Trace:
I20250111 02:57:26.164862 19327 rpcz_store.cc:276] 0111 02:57:22.244079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.244432 (+ 353us) service_pool.cc:225] Handling call
0111 02:57:26.083930 (+3839498us) ts_tablet_manager.cc:1145] Deleting tablet 1b2377893bfc426fa00d5ddacd9c5961
0111 02:57:26.083997 (+ 67us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:26.118943 (+ 34946us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:26.144953 (+ 26010us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:26.164403 (+ 19450us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3838579}
I20250111 02:57:26.166195 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 1b2377893bfc426fa00d5ddacd9c5961 (table default.Table_03f0b101f0c64274a40b43897fe6dcdb [id=d5b01ba62b1a42e7a1dca7769742db0d]) successfully deleted
I20250111 02:57:26.172637 19327 tablet_replica.cc:331] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:26.173435 19327 raft_consensus.cc:2238] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:26.174974 19327 raft_consensus.cc:2267] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:26.179106 19327 ts_tablet_manager.cc:1905] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:26.198292 19757 consensus_queue.cc:1035] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:26.207669 19330 tablet_bootstrap.cc:492] T 49616e4e7cc647509e72cde57b87d6fa P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:26.208593 19330 ts_tablet_manager.cc:1397] T 49616e4e7cc647509e72cde57b87d6fa P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.280s user 0.011s sys 0.002s
I20250111 02:57:26.211846 19321 ts_tablet_manager.cc:1428] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.066s user 0.006s sys 0.001s
I20250111 02:57:26.212833 19321 tablet_bootstrap.cc:492] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:26.218947 19327 ts_tablet_manager.cc:1918] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:26.221719 19330 raft_consensus.cc:357] T 49616e4e7cc647509e72cde57b87d6fa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.222518 19330 raft_consensus.cc:383] T 49616e4e7cc647509e72cde57b87d6fa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:26.222801 19330 raft_consensus.cc:738] T 49616e4e7cc647509e72cde57b87d6fa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:26.223718 19327 log.cc:1198] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/0267a92995ea460786dc26813b5895cf
I20250111 02:57:26.224519 19327 ts_tablet_manager.cc:1939] T 0267a92995ea460786dc26813b5895cf P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:26.226534 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 144) took 3517 ms. Trace:
I20250111 02:57:26.226943 19327 rpcz_store.cc:276] 0111 02:57:22.709235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.709379 (+ 144us) service_pool.cc:225] Handling call
0111 02:57:26.172452 (+3463073us) ts_tablet_manager.cc:1145] Deleting tablet 0267a92995ea460786dc26813b5895cf
0111 02:57:26.172512 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:26.194691 (+ 22179us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:26.218719 (+ 24028us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:26.226471 (+ 7752us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3462487}
I20250111 02:57:26.228441 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 0267a92995ea460786dc26813b5895cf (table default.Table_62556c9dbbcb4e8a83e01aef82e41449 [id=e7ade2345b0845acb3ecdaa6f1ff6e7b]) successfully deleted
I20250111 02:57:26.230001 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 145) took 3208 ms. Trace:
I20250111 02:57:26.230392 19327 rpcz_store.cc:276] 0111 02:57:23.021480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:23.021652 (+ 172us) service_pool.cc:225] Handling call
0111 02:57:26.229508 (+3207856us) ts_tablet_manager.cc:1145] Deleting tablet 3052e9707de14e1f9e38236d56908ed3
0111 02:57:26.229587 (+ 79us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:26.229922 (+ 335us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3207106}
W20250111 02:57:26.231828 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 3052e9707de14e1f9e38236d56908ed3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3052e9707de14e1f9e38236d56908ed3 already in progress: creating tablet
I20250111 02:57:26.233109 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 146) took 2851 ms. Trace:
I20250111 02:57:26.233505 19327 rpcz_store.cc:276] 0111 02:57:23.381851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:23.411556 (+ 29705us) service_pool.cc:225] Handling call
0111 02:57:26.232693 (+2821137us) ts_tablet_manager.cc:1145] Deleting tablet b26e5db463e94d919640a07a2f8574a8
0111 02:57:26.232762 (+ 69us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:26.233057 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2776222}
W20250111 02:57:26.234815 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet b26e5db463e94d919640a07a2f8574a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b26e5db463e94d919640a07a2f8574a8 already in progress: creating tablet
I20250111 02:57:26.236068 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 147) took 1847 ms. Trace:
I20250111 02:57:26.236428 19327 rpcz_store.cc:276] 0111 02:57:24.394074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:24.396522 (+ 2448us) service_pool.cc:225] Handling call
0111 02:57:26.235637 (+1839115us) ts_tablet_manager.cc:1145] Deleting tablet 0ef86e1fd412484db83e7d39c9059a90
0111 02:57:26.235707 (+ 70us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:26.235993 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1838512}
W20250111 02:57:26.237730 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 0ef86e1fd412484db83e7d39c9059a90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ef86e1fd412484db83e7d39c9059a90 already in progress: creating tablet
I20250111 02:57:26.238793 19327 tablet_replica.cc:331] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:26.239580 19327 raft_consensus.cc:2238] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:57:26.240805 19327 raft_consensus.cc:2267] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:26.243013 19327 ts_tablet_manager.cc:1905] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:26.243986 19585 raft_consensus.cc:491] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:26.244434 19585 raft_consensus.cc:513] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.246374 19585 leader_election.cc:290] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:26.248275 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b69a8c2caea4bddac3f539733bcd3af" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:26.249760 19133 leader_election.cc:343] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:26.251410 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b69a8c2caea4bddac3f539733bcd3af" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:26.275158 19330 consensus_queue.cc:260] T 49616e4e7cc647509e72cde57b87d6fa P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:26.275689 19131 leader_election.cc:343] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:26.276114 19131 leader_election.cc:304] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:26.279407 19696 raft_consensus.cc:2743] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:26.279932 19330 ts_tablet_manager.cc:1428] T 49616e4e7cc647509e72cde57b87d6fa P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.071s user 0.006s sys 0.001s
I20250111 02:57:26.280771 19330 tablet_bootstrap.cc:492] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:26.288453 19244 heartbeater.cc:502] Master 127.18.20.126:45115 was elected leader, sending a full tablet report...
I20250111 02:57:26.289407 19321 tablet_bootstrap.cc:654] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:26.298462 19330 tablet_bootstrap.cc:654] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:26.305492 19327 ts_tablet_manager.cc:1918] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250111 02:57:26.305879 19327 log.cc:1198] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/b8e8c72c99ac4c369957b571d74baffe
I20250111 02:57:26.306054 19322 tablet_bootstrap.cc:654] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:26.306603 19327 ts_tablet_manager.cc:1939] T b8e8c72c99ac4c369957b571d74baffe P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:26.306783 19326 ts_tablet_manager.cc:1918] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:26.307193 19326 log.cc:1198] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/1b2377893bfc426fa00d5ddacd9c5961
I20250111 02:57:26.307929 19326 ts_tablet_manager.cc:1939] T 1b2377893bfc426fa00d5ddacd9c5961 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:26.308900 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 153) took 1800 ms. Trace:
I20250111 02:57:26.309211 19327 rpcz_store.cc:276] 0111 02:57:24.508822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:24.511146 (+ 2324us) service_pool.cc:225] Handling call
0111 02:57:26.238625 (+1727479us) ts_tablet_manager.cc:1145] Deleting tablet b8e8c72c99ac4c369957b571d74baffe
0111 02:57:26.238683 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:26.265328 (+ 26645us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:26.280759 (+ 15431us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:26.308820 (+ 28061us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1726853}
I20250111 02:57:26.309942 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 148) took 4067 ms. Trace:
I20250111 02:57:26.310575 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 154) took 1043 ms. Trace:
I20250111 02:57:26.310566 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet b8e8c72c99ac4c369957b571d74baffe (table default.Table_35d7f864912b40bcb5251a90d40c7849 [id=772d994bee2d4e34bd48912acecf8d48]) successfully deleted
I20250111 02:57:26.310252 19326 rpcz_store.cc:276] 0111 02:57:22.242522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.242694 (+ 172us) service_pool.cc:225] Handling call
0111 02:57:25.966965 (+3724271us) ts_tablet_manager.cc:1145] Deleting tablet 1b2377893bfc426fa00d5ddacd9c5961
0111 02:57:25.967012 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:26.165724 (+198712us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:26.306553 (+140829us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:26.309876 (+ 3323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3723543}
I20250111 02:57:26.311493 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 1b2377893bfc426fa00d5ddacd9c5961 (table default.Table_03f0b101f0c64274a40b43897fe6dcdb [id=d5b01ba62b1a42e7a1dca7769742db0d]) successfully deleted
I20250111 02:57:26.321362 19321 tablet_bootstrap.cc:492] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:26.321816 19321 ts_tablet_manager.cc:1397] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.109s user 0.008s sys 0.002s
I20250111 02:57:26.325021 19321 raft_consensus.cc:357] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.325726 19321 raft_consensus.cc:383] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:26.325979 19321 raft_consensus.cc:738] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:26.328549 19321 consensus_queue.cc:260] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.329623 19330 tablet_bootstrap.cc:492] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:26.333284 19330 ts_tablet_manager.cc:1397] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.005s
I20250111 02:57:26.336321 19330 raft_consensus.cc:357] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.338666 19321 ts_tablet_manager.cc:1428] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250111 02:57:26.339628 19321 tablet_bootstrap.cc:492] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:26.337014 19330 raft_consensus.cc:383] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:26.342512 19330 raft_consensus.cc:738] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:26.343261 19330 consensus_queue.cc:260] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.346726 19322 tablet_bootstrap.cc:492] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:26.347218 19330 ts_tablet_manager.cc:1428] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20250111 02:57:26.347276 19322 ts_tablet_manager.cc:1397] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.218s user 0.011s sys 0.016s
I20250111 02:57:26.348125 19330 tablet_bootstrap.cc:492] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:26.351415 19322 raft_consensus.cc:357] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:26.352229 19322 raft_consensus.cc:383] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:26.353978 19322 raft_consensus.cc:738] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:26.310868 19327 rpcz_store.cc:276] 0111 02:57:25.267035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:25.267221 (+ 186us) service_pool.cc:225] Handling call
0111 02:57:26.310152 (+1042931us) ts_tablet_manager.cc:1145] Deleting tablet 74592f80afab4513a9eb2eac351c4de5
0111 02:57:26.310220 (+ 68us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:26.310507 (+ 287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1042406}
W20250111 02:57:26.355382 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 74592f80afab4513a9eb2eac351c4de5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74592f80afab4513a9eb2eac351c4de5 already in progress: creating tablet
I20250111 02:57:26.356510 19322 consensus_queue.cc:260] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:26.360553 18617 catalog_manager.cc:4839] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet b54311f4269d4575b8998d7f545000e7 because the tablet was not found. No further retry: Not found: Tablet not found: b54311f4269d4575b8998d7f545000e7
W20250111 02:57:26.367625 18617 catalog_manager.cc:4839] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 29561e984e6d40fea032075f0531dd1f because the tablet was not found. No further retry: Not found: Tablet not found: 29561e984e6d40fea032075f0531dd1f
I20250111 02:57:26.367722 18624 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 061e6eb5775443679e9cbd6b11f66fe3
I20250111 02:57:26.369544 18624 catalog_manager.cc:6596] ReplaceTablet: tablet 061e6eb5775443679e9cbd6b11f66fe3 deleted and replaced by tablet 218b87091d9546ba859f954a8f9e089e
I20250111 02:57:26.370716 19023 tablet_service.cc:1514] Processing DeleteTablet for tablet 061e6eb5775443679e9cbd6b11f66fe3 with delete_type TABLET_DATA_DELETED (replaced by tablet 218b87091d9546ba859f954a8f9e089e) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:26.372023 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 061e6eb5775443679e9cbd6b11f66fe3 with delete_type TABLET_DATA_DELETED (replaced by tablet 218b87091d9546ba859f954a8f9e089e) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:26.373538 19322 ts_tablet_manager.cc:1428] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.026s user 0.003s sys 0.003s
W20250111 02:57:26.374825 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 061e6eb5775443679e9cbd6b11f66fe3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 061e6eb5775443679e9cbd6b11f66fe3 already in progress: creating tablet
I20250111 02:57:26.380956 19322 tablet_bootstrap.cc:492] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:26.394579 18621 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 42230a0bc94249fda91cf79fba48f024
I20250111 02:57:26.396265 19321 tablet_bootstrap.cc:654] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:26.401165 19696 consensus_queue.cc:1035] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:26.405200 19322 tablet_bootstrap.cc:654] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:26.406335 19330 tablet_bootstrap.cc:654] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:26.408413 19177 tablet_service.cc:1514] Processing DeleteTablet for tablet 061e6eb5775443679e9cbd6b11f66fe3 with delete_type TABLET_DATA_DELETED (replaced by tablet 218b87091d9546ba859f954a8f9e089e) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:26.409121 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet 42230a0bc94249fda91cf79fba48f024 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:24 UTC) from {username='slave'} at 127.0.0.1:32868
W20250111 02:57:26.442409 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 061e6eb5775443679e9cbd6b11f66fe3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 061e6eb5775443679e9cbd6b11f66fe3 already in progress: creating tablet
W20250111 02:57:26.443235 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 42230a0bc94249fda91cf79fba48f024 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42230a0bc94249fda91cf79fba48f024 already in progress: creating tablet
I20250111 02:57:26.446156 19023 tablet_service.cc:1514] Processing DeleteTablet for tablet 42230a0bc94249fda91cf79fba48f024 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:24 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:26.446910 18621 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47522 (request call id 229) took 1783 ms. Trace:
I20250111 02:57:26.449272 19696 consensus_queue.cc:1035] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:26.450580 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 42230a0bc94249fda91cf79fba48f024 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:24 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:26.451443 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_d7e00ace61e545129a2e99093dba16db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:26.452199 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 42230a0bc94249fda91cf79fba48f024 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42230a0bc94249fda91cf79fba48f024 already in progress: creating tablet
W20250111 02:57:26.452965 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7e00ace61e545129a2e99093dba16db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:26.469897 18627 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet 199ca8692f39445f93d8c27e2134a32c (table default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]). Expected version 2 got 0
W20250111 02:57:26.472643 19133 consensus_peers.cc:487] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:26.447266 18621 rpcz_store.cc:276] 0111 02:57:24.663687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:24.663880 (+ 193us) service_pool.cc:225] Handling call
0111 02:57:24.665292 (+ 1412us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0111 02:57:24.666234 (+ 942us) catalog_manager.cc:2753] Modifying in-memory table state
0111 02:57:24.666385 (+ 151us) catalog_manager.cc:2762] Locking tablets
0111 02:57:26.323282 (+1656897us) catalog_manager.cc:2776] Removing table and tablets from system table
0111 02:57:26.393671 (+ 70389us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:26.393702 (+ 31us) write_op.cc:454] Released schema lock
0111 02:57:26.394008 (+ 306us) catalog_manager.cc:2794] Removing table from by-name map
0111 02:57:26.394309 (+ 301us) catalog_manager.cc:2810] Committing in-memory state
0111 02:57:26.394463 (+ 154us) catalog_manager.cc:2814] Aborting table tasks
0111 02:57:26.446844 (+ 52381us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:26.325286 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:26.325642 (+ 356us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:26.325667 (+ 25us) write_op.cc:435] Acquired schema lock
0111 02:57:26.325680 (+ 13us) tablet.cc:659] Decoding operations
0111 02:57:26.325986 (+ 306us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:26.326032 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:26.326055 (+ 23us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:26.326318 (+ 263us) tablet.cc:698] Row locks acquired
0111 02:57:26.326332 (+ 14us) write_op.cc:260] PREPARE: finished
0111 02:57:26.326441 (+ 109us) write_op.cc:270] Start()
0111 02:57:26.326528 (+ 87us) write_op.cc:276] Timestamp: P: 1736564246326297 usec, L: 0
0111 02:57:26.326543 (+ 15us) op_driver.cc:348] REPLICATION: starting
0111 02:57:26.327035 (+ 492us) log.cc:844] Serialized 728 byte log entry
0111 02:57:26.359222 (+ 32187us) op_driver.cc:464] REPLICATION: finished
0111 02:57:26.360180 (+ 958us) write_op.cc:301] APPLY: starting
0111 02:57:26.360257 (+ 77us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:26.360413 (+ 156us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:26.360540 (+ 127us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:26.360832 (+ 292us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:26.361014 (+ 182us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0111 02:57:26.361070 (+ 56us) write_op.cc:312] APPLY: finished
0111 02:57:26.361323 (+ 253us) log.cc:844] Serialized 29 byte log entry
0111 02:57:26.361492 (+ 169us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:26.361594 (+ 102us) write_op.cc:454] Released schema lock
0111 02:57:26.361659 (+ 65us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":764,"apply.run_cpu_time_us":1689,"apply.run_wall_time_us":1689,"num_ops":2,"prepare.queue_time_us":114,"prepare.run_cpu_time_us":2094,"prepare.run_wall_time_us":2095,"raft.queue_time_us":3938,"raft.run_cpu_time_us":1084,"raft.run_wall_time_us":1084,"replication_time_us":32542}]]}
I20250111 02:57:26.484252 18625 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet 40fdec982cf643c68ef90798f4f65ad4 (table default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927]). Expected version 1 got 0
I20250111 02:57:26.484854 18625 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]). Expected version 2 got 0
I20250111 02:57:26.485383 18890 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb87813c28742edad99470c84269ef4 (DEFAULT_TABLE table=default.Table_f9181ca352384593b93119fb3e0cd812 [id=e0ac9d601c464a3daaaa63c4cb90e471]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.486919 18890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eb87813c28742edad99470c84269ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.491461 19321 tablet_bootstrap.cc:492] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:26.491917 19321 ts_tablet_manager.cc:1397] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.152s user 0.013s sys 0.003s
I20250111 02:57:26.494618 19321 raft_consensus.cc:357] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.495355 19321 raft_consensus.cc:383] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:26.495625 19321 raft_consensus.cc:738] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:26.498685 18884 tablet_service.cc:1514] Processing DeleteTablet for tablet ef70b417568049f0b496f4d8e5f7e0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7ea78feee64340e39404cba0f1bf04db) from {username='slave'} at 127.0.0.1:51082
W20250111 02:57:26.500485 18617 catalog_manager.cc:4839] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet ef70b417568049f0b496f4d8e5f7e0e4 because the tablet was not found. No further retry: Not found: Tablet not found: ef70b417568049f0b496f4d8e5f7e0e4
I20250111 02:57:26.500530 19754 consensus_queue.cc:1035] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:26.502789 18884 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e8c72c99ac4c369957b571d74baffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:23 UTC) from {username='slave'} at 127.0.0.1:51082
W20250111 02:57:26.504567 18617 catalog_manager.cc:4839] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet b8e8c72c99ac4c369957b571d74baffe because the tablet was not found. No further retry: Not found: Tablet not found: b8e8c72c99ac4c369957b571d74baffe
I20250111 02:57:26.511776 19244 heartbeater.cc:502] Master 127.18.20.126:45115 was elected leader, sending a full tablet report...
I20250111 02:57:26.515976 19023 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb87813c28742edad99470c84269ef4 (DEFAULT_TABLE table=default.Table_f9181ca352384593b93119fb3e0cd812 [id=e0ac9d601c464a3daaaa63c4cb90e471]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.516577 19177 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb87813c28742edad99470c84269ef4 (DEFAULT_TABLE table=default.Table_f9181ca352384593b93119fb3e0cd812 [id=e0ac9d601c464a3daaaa63c4cb90e471]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.517923 19177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eb87813c28742edad99470c84269ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.519641 18884 tablet_service.cc:1467] Processing CreateTablet for tablet aca35147034446b3882b5244923bddc7 (DEFAULT_TABLE table=default.Table_1431a031eea3496b9764a373a012a367 [id=a0cdf8fe647d4c84a9ce97df9ce9a0ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.520151 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet 92aa64b2de5746d1b12cc93a10c9d56e with delete_type TABLET_DATA_DELETED (replaced by tablet e46894710a72407993b36801f3b90eb0) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:26.521181 19327 tablet_replica.cc:331] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:26.521349 18884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aca35147034446b3882b5244923bddc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.521871 19327 raft_consensus.cc:2238] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:26.522529 19327 pending_rounds.cc:62] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c: Trying to abort 1 pending ops.
I20250111 02:57:26.522775 19327 pending_rounds.cc:69] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112967150791327744 op_type: NO_OP noop_request { }
I20250111 02:57:26.523406 19327 raft_consensus.cc:2883] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:57:26.523730 19327 raft_consensus.cc:2267] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:26.525892 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2377893bfc426fa00d5ddacd9c5961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:21 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:26.526149 19327 ts_tablet_manager.cc:1905] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:26.527288 18625 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.65:43151 (request call id 53) took 985 ms (client timeout 1000 ms). Trace:
W20250111 02:57:26.527699 18625 rpcz_store.cc:269] 0111 02:57:25.541942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:25.542072 (+ 130us) service_pool.cc:225] Handling call
0111 02:57:26.527187 (+985115us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":6}
I20250111 02:57:26.514881 19321 consensus_queue.cc:260] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.536904 19023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eb87813c28742edad99470c84269ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.537918 19754 raft_consensus.cc:491] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:26.538456 19754 raft_consensus.cc:513] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:26.539568 19326 tablet_replica.cc:331] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:26.540385 19326 raft_consensus.cc:2238] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:26.540735 19754 leader_election.cc:290] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:26.542460 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea78feee64340e39404cba0f1bf04db" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:26.544116 19133 leader_election.cc:343] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:26.544076 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea78feee64340e39404cba0f1bf04db" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:26.545856 19131 leader_election.cc:343] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:26.546248 19131 leader_election.cc:304] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:26.546928 19754 raft_consensus.cc:2743] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:26.549374 19321 ts_tablet_manager.cc:1428] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.057s user 0.006s sys 0.000s
I20250111 02:57:26.540884 19326 raft_consensus.cc:2267] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:26.554839 19327 ts_tablet_manager.cc:1918] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:26.555382 19327 log.cc:1198] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/92aa64b2de5746d1b12cc93a10c9d56e
I20250111 02:57:26.556149 19327 ts_tablet_manager.cc:1939] T 92aa64b2de5746d1b12cc93a10c9d56e P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
W20250111 02:57:26.559108 19131 consensus_peers.cc:487] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa -> Peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Couldn't send request to peer 54628fbcec2f4359aa9e26a734d4d29f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:26.559582 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 92aa64b2de5746d1b12cc93a10c9d56e (table default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3]) successfully deleted
W20250111 02:57:26.562539 18617 catalog_manager.cc:4839] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 1b2377893bfc426fa00d5ddacd9c5961 because the tablet was not found. No further retry: Not found: Tablet not found: 1b2377893bfc426fa00d5ddacd9c5961
I20250111 02:57:26.574028 18621 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_292c103f235f421db8dacf834cca2c53" } new_table_name: "default.Table_83e6031ed7fe4cbe88502007987425a1" modify_external_catalogs: true
I20250111 02:57:26.575579 18629 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 48dd0bccf6854d37a7ef95b5ab06308e
I20250111 02:57:26.576929 18629 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47522 (request call id 230) took 1894 ms. Trace:
I20250111 02:57:26.577899 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet 48dd0bccf6854d37a7ef95b5ab06308e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:24 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:26.578274 19326 ts_tablet_manager.cc:1905] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:26.581362 19028 tablet_service.cc:1514] Processing DeleteTablet for tablet 48dd0bccf6854d37a7ef95b5ab06308e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:24 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:26.583820 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet 48dd0bccf6854d37a7ef95b5ab06308e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:24 UTC) from {username='slave'} at 127.0.0.1:51082
W20250111 02:57:26.585294 18617 catalog_manager.cc:4839] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 48dd0bccf6854d37a7ef95b5ab06308e because the tablet was not found. No further retry: Not found: Tablet not found: 48dd0bccf6854d37a7ef95b5ab06308e
I20250111 02:57:26.586658 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_bdf982e1d6904892ac08ff16ed20a636"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:26.588323 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdf982e1d6904892ac08ff16ed20a636 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:26.577232 18629 rpcz_store.cc:276] 0111 02:57:24.682731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:24.757516 (+ 74785us) service_pool.cc:225] Handling call
0111 02:57:24.758895 (+ 1379us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0111 02:57:24.759471 (+ 576us) catalog_manager.cc:2753] Modifying in-memory table state
0111 02:57:24.759614 (+ 143us) catalog_manager.cc:2762] Locking tablets
0111 02:57:26.447594 (+1687980us) catalog_manager.cc:2776] Removing table and tablets from system table
0111 02:57:26.575171 (+127577us) catalog_manager.cc:2794] Removing table from by-name map
0111 02:57:26.575378 (+ 207us) catalog_manager.cc:2810] Committing in-memory state
0111 02:57:26.575482 (+ 104us) catalog_manager.cc:2814] Aborting table tasks
0111 02:57:26.576869 (+ 1387us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:26.459738 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:26.460094 (+ 356us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:26.460118 (+ 24us) write_op.cc:435] Acquired schema lock
0111 02:57:26.460132 (+ 14us) tablet.cc:659] Decoding operations
0111 02:57:26.460445 (+ 313us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:26.460487 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:26.460510 (+ 23us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:26.460760 (+ 250us) tablet.cc:698] Row locks acquired
0111 02:57:26.460772 (+ 12us) write_op.cc:260] PREPARE: finished
0111 02:57:26.460879 (+ 107us) write_op.cc:270] Start()
0111 02:57:26.460958 (+ 79us) write_op.cc:276] Timestamp: P: 1736564246460738 usec, L: 0
0111 02:57:26.460972 (+ 14us) op_driver.cc:348] REPLICATION: starting
0111 02:57:26.461504 (+ 532us) log.cc:844] Serialized 759 byte log entry
0111 02:57:26.571428 (+109924us) op_driver.cc:464] REPLICATION: finished
0111 02:57:26.571770 (+ 342us) write_op.cc:301] APPLY: starting
0111 02:57:26.571828 (+ 58us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:26.571950 (+ 122us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:26.572061 (+ 111us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:26.572307 (+ 246us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:26.572458 (+ 151us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0111 02:57:26.572512 (+ 54us) write_op.cc:312] APPLY: finished
0111 02:57:26.572701 (+ 189us) log.cc:844] Serialized 29 byte log entry
0111 02:57:26.572849 (+ 148us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:26.572910 (+ 61us) write_op.cc:454] Released schema lock
0111 02:57:26.572945 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:26.573177 (+ 232us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:26.573200 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":165,"apply.run_cpu_time_us":1393,"apply.run_wall_time_us":1394,"num_ops":2,"prepare.queue_time_us":10219,"prepare.run_cpu_time_us":2045,"prepare.run_wall_time_us":2045,"raft.queue_time_us":1172,"raft.run_cpu_time_us":883,"raft.run_wall_time_us":1203,"replication_time_us":110348}]]}
I20250111 02:57:26.596036 18629 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_6a0ed66485f640e9a2c11b62b5cbd239"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:26.597842 18629 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a0ed66485f640e9a2c11b62b5cbd239 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:26.605468 19321 tablet_bootstrap.cc:492] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
W20250111 02:57:26.605787 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 48dd0bccf6854d37a7ef95b5ab06308e because the tablet was not found. No further retry: Not found: Tablet not found: 48dd0bccf6854d37a7ef95b5ab06308e
I20250111 02:57:26.606910 19330 tablet_bootstrap.cc:492] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:26.607460 19330 ts_tablet_manager.cc:1397] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.260s user 0.009s sys 0.010s
I20250111 02:57:26.610316 19330 raft_consensus.cc:357] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:26.625718 19133 consensus_peers.cc:487] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:26.629855 19174 tablet_service.cc:1467] Processing CreateTablet for tablet aca35147034446b3882b5244923bddc7 (DEFAULT_TABLE table=default.Table_1431a031eea3496b9764a373a012a367 [id=a0cdf8fe647d4c84a9ce97df9ce9a0ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.631100 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aca35147034446b3882b5244923bddc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.643024 19321 tablet_bootstrap.cc:654] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:26.640517 19028 tablet_service.cc:1467] Processing CreateTablet for tablet aca35147034446b3882b5244923bddc7 (DEFAULT_TABLE table=default.Table_1431a031eea3496b9764a373a012a367 [id=a0cdf8fe647d4c84a9ce97df9ce9a0ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.656062 19696 raft_consensus.cc:491] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:26.656690 19696 raft_consensus.cc:513] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.660346 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49616e4e7cc647509e72cde57b87d6fa" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:26.661724 19322 tablet_bootstrap.cc:492] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:26.662410 19322 ts_tablet_manager.cc:1397] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.282s user 0.002s sys 0.014s
W20250111 02:57:26.662405 19133 leader_election.cc:343] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:26.666823 19131 consensus_peers.cc:487] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa -> Peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Couldn't send request to peer 54628fbcec2f4359aa9e26a734d4d29f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b2377893bfc426fa00d5ddacd9c5961. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:26.667404 19696 leader_election.cc:290] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:26.667343 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49616e4e7cc647509e72cde57b87d6fa" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:26.668094 19051 raft_consensus.cc:2463] T 49616e4e7cc647509e72cde57b87d6fa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:26.669350 19131 leader_election.cc:304] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:26.670606 19696 raft_consensus.cc:2798] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:26.671092 19696 raft_consensus.cc:491] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:26.671636 19696 raft_consensus.cc:3054] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:26.675554 18614 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b4273afc264843029ed0baf1fc5c8a3e (table default.Table_4e3a79a5d2184687a8c51fd2702c5fd0 [id=73c59c0d9864497fbc9f19e265d8283f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b4273afc264843029ed0baf1fc5c8a3e consensus state has no leader
I20250111 02:57:26.679553 19330 raft_consensus.cc:383] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:26.680037 19330 raft_consensus.cc:738] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:26.680104 19696 raft_consensus.cc:513] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.680859 19330 consensus_queue.cc:260] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.684968 18623 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 199ca8692f39445f93d8c27e2134a32c
I20250111 02:57:26.685079 19321 tablet_bootstrap.cc:492] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:26.685539 19321 ts_tablet_manager.cc:1397] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.080s user 0.009s sys 0.003s
I20250111 02:57:26.686986 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet 199ca8692f39445f93d8c27e2134a32c with delete_type TABLET_DATA_DELETED (replaced by tablet 29f27ed2c53d43d7a51ee2e45afaf9fc) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:26.687546 19029 tablet_service.cc:1514] Processing DeleteTablet for tablet 199ca8692f39445f93d8c27e2134a32c with delete_type TABLET_DATA_DELETED (replaced by tablet 29f27ed2c53d43d7a51ee2e45afaf9fc) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:26.687012 19602 raft_consensus.cc:491] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:26.688891 19602 raft_consensus.cc:513] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:26.691277 19177 tablet_service.cc:1467] Processing CreateTablet for tablet 36f71f8dd2894d40b3bfdd20c1453787 (DEFAULT_TABLE table=default.Table_cce40438ec8e4527b01088af6b7b0981 [id=cb42d3930a9a44bfb5dc40feb979d505]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.692059 19029 tablet_service.cc:1467] Processing CreateTablet for tablet 36f71f8dd2894d40b3bfdd20c1453787 (DEFAULT_TABLE table=default.Table_cce40438ec8e4527b01088af6b7b0981 [id=cb42d3930a9a44bfb5dc40feb979d505]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.692708 19177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f71f8dd2894d40b3bfdd20c1453787. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.689514 18623 catalog_manager.cc:6596] ReplaceTablet: tablet 199ca8692f39445f93d8c27e2134a32c deleted and replaced by tablet 29f27ed2c53d43d7a51ee2e45afaf9fc
I20250111 02:57:26.693234 18623 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47522 (request call id 233) took 1600 ms. Trace:
I20250111 02:57:26.694058 18882 tablet_service.cc:1467] Processing CreateTablet for tablet d6da575f052d4b20a8e41f0bd9dab7e4 (DEFAULT_TABLE table=default.Table_02b5991089a34b0b8ee6039ccc092c93 [id=977f59b134154bc396bf42bc1e44e717]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.695214 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6da575f052d4b20a8e41f0bd9dab7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.693543 18623 rpcz_store.cc:276] 0111 02:57:25.093195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:25.265433 (+172238us) service_pool.cc:225] Handling call
0111 02:57:26.693177 (+1427744us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:26.560685 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:26.561038 (+ 353us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:26.561061 (+ 23us) write_op.cc:435] Acquired schema lock
0111 02:57:26.561073 (+ 12us) tablet.cc:659] Decoding operations
0111 02:57:26.561382 (+ 309us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:26.561417 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:26.561438 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:26.561636 (+ 198us) tablet.cc:698] Row locks acquired
0111 02:57:26.561650 (+ 14us) write_op.cc:260] PREPARE: finished
0111 02:57:26.561759 (+ 109us) write_op.cc:270] Start()
0111 02:57:26.561838 (+ 79us) write_op.cc:276] Timestamp: P: 1736564246561620 usec, L: 0
0111 02:57:26.561851 (+ 13us) op_driver.cc:348] REPLICATION: starting
0111 02:57:26.562346 (+ 495us) log.cc:844] Serialized 625 byte log entry
0111 02:57:26.618153 (+ 55807us) op_driver.cc:464] REPLICATION: finished
0111 02:57:26.631048 (+ 12895us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b60000641c8. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd428348a81 00007fd4284b365c 00007fd428414c92 00007fd4284152b9 00007fd42841754a 00007fd4284174d8 00007fd428417458 00007fd428417413 00007fd42841662c 00007fd429d05bb6 00007fd429d00568
0111 02:57:26.631304 (+ 256us) write_op.cc:301] APPLY: starting
0111 02:57:26.631371 (+ 67us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:26.631510 (+ 139us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:26.631549 (+ 39us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:26.631910 (+ 361us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:26.632141 (+ 231us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:26.632231 (+ 90us) write_op.cc:312] APPLY: finished
0111 02:57:26.632504 (+ 273us) log.cc:844] Serialized 29 byte log entry
0111 02:57:26.632703 (+ 199us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:26.632801 (+ 98us) write_op.cc:454] Released schema lock
0111 02:57:26.632862 (+ 61us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:26.645606 (+ 12744us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b5400021300. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd428348a81 00007fd42841d16d 00007fd4284161f2 00007fd428415983 00007fd428419b91 00007fd428419b49 00007fd428419ad9 00007fd428419aa1 00007fd428418ccd 000000000059ea04 000000000059e839
0111 02:57:26.645812 (+ 206us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:26.645853 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":86,"apply.run_cpu_time_us":3836,"apply.run_wall_time_us":14440,"num_ops":2,"prepare.queue_time_us":80149,"prepare.run_cpu_time_us":1856,"prepare.run_wall_time_us":1854,"replication_time_us":56204,"spinlock_wait_cycles":49379584}]]}
I20250111 02:57:26.691268 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet 199ca8692f39445f93d8c27e2134a32c with delete_type TABLET_DATA_DELETED (replaced by tablet 29f27ed2c53d43d7a51ee2e45afaf9fc) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:26.698812 19833 tablet_replica.cc:331] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:26.699384 19481 raft_consensus.cc:491] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:26.699936 19833 raft_consensus.cc:2238] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:26.699935 19481 raft_consensus.cc:513] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.700506 19833 raft_consensus.cc:2267] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
W20250111 02:57:26.701777 18986 consensus_peers.cc:487] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:26.702426 19481 leader_election.cc:290] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:26.703204 19321 raft_consensus.cc:357] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:26.703895 19321 raft_consensus.cc:383] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:26.704155 19321 raft_consensus.cc:738] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
W20250111 02:57:26.704347 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 199ca8692f39445f93d8c27e2134a32c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 199ca8692f39445f93d8c27e2134a32c already in progress: creating tablet
I20250111 02:57:26.706127 18881 tablet_service.cc:1467] Processing CreateTablet for tablet 36f71f8dd2894d40b3bfdd20c1453787 (DEFAULT_TABLE table=default.Table_cce40438ec8e4527b01088af6b7b0981 [id=cb42d3930a9a44bfb5dc40feb979d505]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.706610 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe8aa967fcb4bd7a48f6f0f69bea496" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:26.707518 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49616e4e7cc647509e72cde57b87d6fa" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:26.707772 18881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f71f8dd2894d40b3bfdd20c1453787. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.708909 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7158a7fc624fa4bc816ac3721c610d" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:26.709625 19197 raft_consensus.cc:2463] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:26.710685 18624 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_d234d85b2b234eef9e672be1cb4a1923" } modify_external_catalogs: true
I20250111 02:57:26.711021 18986 leader_election.cc:304] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:26.711292 18624 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_d234d85b2b234eef9e672be1cb4a1923" } modify_external_catalogs: true
I20250111 02:57:26.711424 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7158a7fc624fa4bc816ac3721c610d" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:26.711813 19815 raft_consensus.cc:2798] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:26.712206 19815 raft_consensus.cc:491] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:26.712172 19321 consensus_queue.cc:260] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:26.712594 19815 raft_consensus.cc:3054] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:26.713263 18989 leader_election.cc:343] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:26.714567 19321 ts_tablet_manager.cc:1428] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20250111 02:57:26.714561 19602 leader_election.cc:290] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:26.702875 19833 ts_tablet_manager.cc:1905] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:26.722517 19133 leader_election.cc:343] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:26.724545 19321 tablet_bootstrap.cc:492] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:26.724993 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe8aa967fcb4bd7a48f6f0f69bea496" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:26.726761 18841 leader_election.cc:304] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:26.728037 19602 raft_consensus.cc:2743] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:26.731904 19028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aca35147034446b3882b5244923bddc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.732909 19029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f71f8dd2894d40b3bfdd20c1453787. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.737205 19481 raft_consensus.cc:491] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:26.737871 19481 raft_consensus.cc:513] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.741458 19696 leader_election.cc:290] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:26.741526 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32afc2f158254811836fcf00f7f409ed" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:26.742508 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49616e4e7cc647509e72cde57b87d6fa" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
W20250111 02:57:26.742966 18989 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:26.743207 19051 raft_consensus.cc:3054] T 49616e4e7cc647509e72cde57b87d6fa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:26.747601 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32afc2f158254811836fcf00f7f409ed" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:26.748576 19197 raft_consensus.cc:2463] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:26.749972 18986 leader_election.cc:304] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:26.750766 19677 raft_consensus.cc:2798] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:26.750942 19481 leader_election.cc:290] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:26.751233 19677 raft_consensus.cc:491] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:26.751681 19677 raft_consensus.cc:3054] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:26.759128 19330 ts_tablet_manager.cc:1428] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.151s user 0.003s sys 0.001s
I20250111 02:57:26.759840 19321 tablet_bootstrap.cc:654] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:26.761072 19330 tablet_bootstrap.cc:492] T f778763e59a94969b3f62be87e2fd867 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:26.767585 19174 tablet_service.cc:1467] Processing CreateTablet for tablet d6da575f052d4b20a8e41f0bd9dab7e4 (DEFAULT_TABLE table=default.Table_02b5991089a34b0b8ee6039ccc092c93 [id=977f59b134154bc396bf42bc1e44e717]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.768633 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6da575f052d4b20a8e41f0bd9dab7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.770169 18880 tablet_service.cc:1467] Processing CreateTablet for tablet e01771c56ea5498a876d3bb865c752a2 (DEFAULT_TABLE table=default.Table_dacea9000ff840659052c41c94a8bfaf [id=51bd1567dc9c43b2877791b72c2c2046]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.771306 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e01771c56ea5498a876d3bb865c752a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.772678 19033 tablet_service.cc:1467] Processing CreateTablet for tablet d6da575f052d4b20a8e41f0bd9dab7e4 (DEFAULT_TABLE table=default.Table_02b5991089a34b0b8ee6039ccc092c93 [id=977f59b134154bc396bf42bc1e44e717]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.773649 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6da575f052d4b20a8e41f0bd9dab7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.777374 19167 tablet_service.cc:1467] Processing CreateTablet for tablet e01771c56ea5498a876d3bb865c752a2 (DEFAULT_TABLE table=default.Table_dacea9000ff840659052c41c94a8bfaf [id=51bd1567dc9c43b2877791b72c2c2046]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.778400 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e01771c56ea5498a876d3bb865c752a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:26.786898 19133 consensus_peers.cc:487] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b2377893bfc426fa00d5ddacd9c5961. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:26.801811 19322 raft_consensus.cc:357] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:26.802537 19322 raft_consensus.cc:383] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:26.802793 19322 raft_consensus.cc:738] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
W20250111 02:57:26.805017 18989 consensus_peers.cc:487] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:26.813055 19244 heartbeater.cc:502] Master 127.18.20.126:45115 was elected leader, sending a full tablet report...
I20250111 02:57:26.816113 19696 raft_consensus.cc:491] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:26.816819 19696 raft_consensus.cc:513] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:26.827606 19322 consensus_queue.cc:260] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:26.830142 19322 ts_tablet_manager.cc:1428] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.167s user 0.001s sys 0.005s
I20250111 02:57:26.827627 19815 raft_consensus.cc:513] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.840272 19833 ts_tablet_manager.cc:1918] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:26.841338 19833 log.cc:1198] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/199ca8692f39445f93d8c27e2134a32c
I20250111 02:57:26.842240 19833 ts_tablet_manager.cc:1939] T 199ca8692f39445f93d8c27e2134a32c P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:26.845263 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845da3ee0863430ea3fe9de8c84f2c17" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:26.847433 19131 leader_election.cc:343] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:26.848338 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 199ca8692f39445f93d8c27e2134a32c (table default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]) successfully deleted
I20250111 02:57:26.848243 19677 raft_consensus.cc:513] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.849248 19696 leader_election.cc:290] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:26.858878 19321 tablet_bootstrap.cc:492] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:26.859428 19321 ts_tablet_manager.cc:1397] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.135s user 0.003s sys 0.012s
I20250111 02:57:26.862385 19321 raft_consensus.cc:357] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.863363 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845da3ee0863430ea3fe9de8c84f2c17" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:26.863827 19322 tablet_bootstrap.cc:492] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
W20250111 02:57:26.864823 19133 leader_election.cc:343] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:26.865252 19133 leader_election.cc:304] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:26.866364 19696 raft_consensus.cc:2743] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:26.866920 19321 raft_consensus.cc:383] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:26.867152 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7158a7fc624fa4bc816ac3721c610d" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:26.867787 19321 raft_consensus.cc:738] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:26.868451 19321 consensus_queue.cc:260] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.876116 19321 ts_tablet_manager.cc:1428] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.016s user 0.004s sys 0.002s
I20250111 02:57:26.892583 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7158a7fc624fa4bc816ac3721c610d" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:26.893319 19197 raft_consensus.cc:3054] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:26.901350 19197 raft_consensus.cc:2463] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
W20250111 02:57:26.911494 18617 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 129058us
I20250111 02:57:26.913142 18623 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_bf5cf724093d4691a60379b87cfab2f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:26.913941 19321 tablet_bootstrap.cc:492] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
W20250111 02:57:26.914568 18614 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 60814.1us
I20250111 02:57:26.920930 19321 tablet_bootstrap.cc:654] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:26.922670 19602 raft_consensus.cc:491] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:26.923238 19602 raft_consensus.cc:513] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:26.924540 19677 leader_election.cc:290] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:26.927042 19602 leader_election.cc:290] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:26.880679 19330 tablet_bootstrap.cc:654] T f778763e59a94969b3f62be87e2fd867 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:26.928999 18841 leader_election.cc:343] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Not found: Tablet not found: 199ca8692f39445f93d8c27e2134a32c
I20250111 02:57:26.884053 19051 raft_consensus.cc:2463] T 49616e4e7cc647509e72cde57b87d6fa P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
W20250111 02:57:26.884621 18989 leader_election.cc:343] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:26.933943 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 57131.3us
I20250111 02:57:26.885461 19815 leader_election.cc:290] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:26.893584 19326 ts_tablet_manager.cc:1918] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:26.941268 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32afc2f158254811836fcf00f7f409ed" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:26.944581 19174 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff06134303d4cb0b4292fe1c0c2d443 (DEFAULT_TABLE table=default.Table_2c181f66f28741b7b388a7b563b7d2f5 [id=ac0e88a97cf740fd8df3e6cbd81888b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.945778 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ff06134303d4cb0b4292fe1c0c2d443. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.947821 19167 tablet_service.cc:1467] Processing CreateTablet for tablet b2f561cfd4f34426b7e42027e53b3ff1 (DEFAULT_TABLE table=default.Table_37a329a4875c463ea093ad5ab23e9cf6 [id=23a0f0c1aaa149c68390a0d0327ff2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.948875 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2f561cfd4f34426b7e42027e53b3ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.948146 18625 catalog_manager.cc:5526] T 40fdec982cf643c68ef90798f4f65ad4 P 8221604733fe4fc0a1fc32de8d6af71c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0cedab82b154d62b78809edc6f28bfa (127.18.20.67). New cstate: current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } }
I20250111 02:57:26.949273 18625 catalog_manager.cc:5618] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535) does not have the latest schema for tablet 40fdec982cf643c68ef90798f4f65ad4 (table default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927]). Expected version 1 got 0
I20250111 02:57:26.937811 18986 leader_election.cc:304] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
W20250111 02:57:26.952962 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 50201.4us
W20250111 02:57:26.953632 18623 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf5cf724093d4691a60379b87cfab2f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:26.954348 18881 tablet_service.cc:1467] Processing CreateTablet for tablet b2f561cfd4f34426b7e42027e53b3ff1 (DEFAULT_TABLE table=default.Table_37a329a4875c463ea093ad5ab23e9cf6 [id=23a0f0c1aaa149c68390a0d0327ff2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.954916 19131 leader_election.cc:304] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:26.955832 18881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2f561cfd4f34426b7e42027e53b3ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.956456 19696 raft_consensus.cc:2798] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:26.957158 19696 raft_consensus.cc:695] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:26.939553 19326 log.cc:1198] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/0267a92995ea460786dc26813b5895cf
I20250111 02:57:26.960824 19326 ts_tablet_manager.cc:1939] T 0267a92995ea460786dc26813b5895cf P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:26.964473 18884 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff06134303d4cb0b4292fe1c0c2d443 (DEFAULT_TABLE table=default.Table_2c181f66f28741b7b388a7b563b7d2f5 [id=ac0e88a97cf740fd8df3e6cbd81888b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.968587 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32afc2f158254811836fcf00f7f409ed" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:26.969240 19197 raft_consensus.cc:3054] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:26.939708 19033 tablet_service.cc:1467] Processing CreateTablet for tablet e01771c56ea5498a876d3bb865c752a2 (DEFAULT_TABLE table=default.Table_dacea9000ff840659052c41c94a8bfaf [id=51bd1567dc9c43b2877791b72c2c2046]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.971948 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e01771c56ea5498a876d3bb865c752a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.973367 19024 tablet_service.cc:1467] Processing CreateTablet for tablet fa86b1e6eca7403c9a9c52592b8fb636 (DEFAULT_TABLE table=default.Table_4b9e85e01e1843ce831205177cbb2e0f [id=0a118da8c5d0408980cbd0b33e0c2ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.974330 19024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa86b1e6eca7403c9a9c52592b8fb636. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.974000 19026 tablet_service.cc:1467] Processing CreateTablet for tablet 64abb30da7fc452cad774bbe59b498f4 (DEFAULT_TABLE table=default.Table_26836b21f0f84973b442a4966f97d1ee [id=0037843fe90a422399b837b548ff45b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.975332 19026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64abb30da7fc452cad774bbe59b498f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.975492 19815 raft_consensus.cc:2798] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:26.976071 19815 raft_consensus.cc:695] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:26.977082 19815 consensus_queue.cc:237] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:26.940649 19028 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff06134303d4cb0b4292fe1c0c2d443 (DEFAULT_TABLE table=default.Table_2c181f66f28741b7b388a7b563b7d2f5 [id=ac0e88a97cf740fd8df3e6cbd81888b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.979943 19023 tablet_service.cc:1467] Processing CreateTablet for tablet b2f561cfd4f34426b7e42027e53b3ff1 (DEFAULT_TABLE table=default.Table_37a329a4875c463ea093ad5ab23e9cf6 [id=23a0f0c1aaa149c68390a0d0327ff2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.980587 19028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ff06134303d4cb0b4292fe1c0c2d443. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:26.948711 18989 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:26.942911 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199ca8692f39445f93d8c27e2134a32c" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:26.983577 18882 tablet_service.cc:1467] Processing CreateTablet for tablet 64abb30da7fc452cad774bbe59b498f4 (DEFAULT_TABLE table=default.Table_26836b21f0f84973b442a4966f97d1ee [id=0037843fe90a422399b837b548ff45b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.984174 19023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2f561cfd4f34426b7e42027e53b3ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.982944 18880 tablet_service.cc:1467] Processing CreateTablet for tablet fa86b1e6eca7403c9a9c52592b8fb636 (DEFAULT_TABLE table=default.Table_4b9e85e01e1843ce831205177cbb2e0f [id=0a118da8c5d0408980cbd0b33e0c2ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.962955 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 150) took 4428 ms. Trace:
I20250111 02:57:26.983505 18890 tablet_service.cc:1467] Processing CreateTablet for tablet d83cfd85a52f438dabe49b6223d5090f (DEFAULT_TABLE table=default.Table_799d0b8328de40a7bacb58ea433fec72 [id=e9efdc6c94574ddca092dd6b9f3b8419]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:26.984939 19326 rpcz_store.cc:276] 0111 02:57:22.534754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.534904 (+ 150us) service_pool.cc:225] Handling call
0111 02:57:26.464465 (+3929561us) ts_tablet_manager.cc:1145] Deleting tablet 0267a92995ea460786dc26813b5895cf
0111 02:57:26.536107 (+ 71642us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:26.793952 (+257845us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:26.893345 (+ 99393us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:26.962887 (+ 69542us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3928876}
I20250111 02:57:26.965981 18884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ff06134303d4cb0b4292fe1c0c2d443. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:26.986162 18989 consensus_peers.cc:487] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:26.986600 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 0267a92995ea460786dc26813b5895cf (table default.Table_62556c9dbbcb4e8a83e01aef82e41449 [id=e7ade2345b0845acb3ecdaa6f1ff6e7b]) successfully deleted
I20250111 02:57:26.987169 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64abb30da7fc452cad774bbe59b498f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.986963 18842 leader_election.cc:304] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:26.987735 19602 raft_consensus.cc:2743] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:26.986195 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 151) took 4394 ms. Trace:
I20250111 02:57:26.988926 19326 rpcz_store.cc:276] 0111 02:57:22.591453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.591572 (+ 119us) service_pool.cc:225] Handling call
0111 02:57:26.985967 (+4394395us) ts_tablet_manager.cc:1145] Deleting tablet 149157c1847a497e88a3136a9c1983e9
0111 02:57:26.986007 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:26.986142 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4393791}
I20250111 02:57:26.989900 19326 tablet_replica.cc:331] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:26.991477 19326 raft_consensus.cc:2238] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:26.991555 18890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d83cfd85a52f438dabe49b6223d5090f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:26.991945 19326 raft_consensus.cc:2267] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:26.993680 19326 ts_tablet_manager.cc:1905] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:26.995536 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa86b1e6eca7403c9a9c52592b8fb636. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:26.999755 18621 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 845da3ee0863430ea3fe9de8c84f2c17 consensus state has no leader
I20250111 02:57:27.000304 18621 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47522 (request call id 243) took 1083 ms. Trace:
I20250111 02:57:26.958267 19696 consensus_queue.cc:237] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:27.015558 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 149157c1847a497e88a3136a9c1983e9 because the tablet was not found. No further retry: Not found: Tablet not found: 149157c1847a497e88a3136a9c1983e9
I20250111 02:57:27.016773 19169 tablet_service.cc:1467] Processing CreateTablet for tablet 64abb30da7fc452cad774bbe59b498f4 (DEFAULT_TABLE table=default.Table_26836b21f0f84973b442a4966f97d1ee [id=0037843fe90a422399b837b548ff45b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.016577 19177 tablet_service.cc:1467] Processing CreateTablet for tablet fa86b1e6eca7403c9a9c52592b8fb636 (DEFAULT_TABLE table=default.Table_4b9e85e01e1843ce831205177cbb2e0f [id=0a118da8c5d0408980cbd0b33e0c2ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.017784 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64abb30da7fc452cad774bbe59b498f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.000594 18621 rpcz_store.cc:276] 0111 02:57:25.917312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:26.507305 (+589993us) service_pool.cc:225] Handling call
0111 02:57:26.573918 (+ 66613us) spinlock_profiling.cc:243] Waited 63.4 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d53986 00007fd427ac0644 00007fd427ac05d2 00007fd427ac0501 00007fd427ac047c 00007fd427abf732 00007fd425d79dfc 00007fd425d79236
0111 02:57:26.574855 (+ 937us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0111 02:57:26.575516 (+ 661us) catalog_manager.cc:3627] Apply alter schema
0111 02:57:26.576051 (+ 535us) catalog_manager.cc:3659] Acquired catalog manager lock
0111 02:57:26.576221 (+ 170us) catalog_manager.cc:3817] Updating metadata on disk
0111 02:57:26.999197 (+422976us) catalog_manager.cc:3848] Committing alterations to in-memory state
0111 02:57:27.000242 (+ 1045us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:26.803721 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:26.804093 (+ 372us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:26.804121 (+ 28us) write_op.cc:435] Acquired schema lock
0111 02:57:26.804134 (+ 13us) tablet.cc:659] Decoding operations
0111 02:57:26.804407 (+ 273us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:26.804445 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:26.804466 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:26.804657 (+ 191us) tablet.cc:698] Row locks acquired
0111 02:57:26.804671 (+ 14us) write_op.cc:260] PREPARE: finished
0111 02:57:26.804781 (+ 110us) write_op.cc:270] Start()
0111 02:57:26.804864 (+ 83us) write_op.cc:276] Timestamp: P: 1736564246804713 usec, L: 0
0111 02:57:26.804878 (+ 14us) op_driver.cc:348] REPLICATION: starting
0111 02:57:26.918122 (+113244us) log.cc:844] Serialized 405 byte log entry
0111 02:57:26.930033 (+ 11911us) spinlock_profiling.cc:243] Waited 5.58 ms on lock 0x7b58000134b0. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427f26a94 00007fd427f2b9d1 00007fd427f2b959 00007fd427f2b8e9 00007fd427f2b8b1 00007fd427f2aa7d 000000000059ea04 000000000059e839 00007fd421064156 00007fd4210675b1
0111 02:57:26.995490 (+ 65457us) op_driver.cc:464] REPLICATION: finished
0111 02:57:26.996270 (+ 780us) write_op.cc:301] APPLY: starting
0111 02:57:26.996329 (+ 59us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:26.996456 (+ 127us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:26.996481 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:26.996682 (+ 201us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:26.996867 (+ 185us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:26.996920 (+ 53us) write_op.cc:312] APPLY: finished
0111 02:57:26.997133 (+ 213us) log.cc:844] Serialized 23 byte log entry
0111 02:57:26.997348 (+ 215us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:26.997424 (+ 76us) write_op.cc:454] Released schema lock
0111 02:57:26.997461 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:26.997762 (+ 301us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:26.997816 (+ 54us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":145814272,"child_traces":[["op",{"apply.queue_time_us":577,"apply.run_cpu_time_us":1442,"apply.run_wall_time_us":1443,"num_ops":1,"prepare.queue_time_us":148462,"prepare.run_cpu_time_us":2238,"prepare.run_wall_time_us":115260,"raft.queue_time_us":4353,"raft.run_cpu_time_us":1394,"raft.run_wall_time_us":6792,"replication_time_us":190500,"spinlock_wait_cycles":13979776}]]}
I20250111 02:57:27.020226 19177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa86b1e6eca7403c9a9c52592b8fb636. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.024276 19677 consensus_queue.cc:1035] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:27.044816 19330 tablet_bootstrap.cc:492] T f778763e59a94969b3f62be87e2fd867 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:27.046061 18908 raft_consensus.cc:3054] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:27.057148 19197 raft_consensus.cc:2463] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:27.060935 18986 leader_election.cc:304] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:27.062196 19677 raft_consensus.cc:2798] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:27.062330 19330 ts_tablet_manager.cc:1397] T f778763e59a94969b3f62be87e2fd867 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.302s user 0.013s sys 0.010s
I20250111 02:57:27.063015 19677 raft_consensus.cc:695] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:27.063931 19677 consensus_queue.cc:237] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:27.072014 18622 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]). Expected version 2 got 0
I20250111 02:57:27.072968 18622 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 40fdec982cf643c68ef90798f4f65ad4 (table default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927]). Expected version 1 got 0
I20250111 02:57:27.085889 18626 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]). Expected version 2 got 0
I20250111 02:57:27.094662 18626 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet a983a670c2104398ac1a159ac3a1da7c (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19]). Expected version 4 got 0
I20250111 02:57:27.095489 18626 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet 40fdec982cf643c68ef90798f4f65ad4 (table default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927]). Expected version 1 got 0
I20250111 02:57:27.095798 19321 tablet_bootstrap.cc:492] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:27.096217 19321 ts_tablet_manager.cc:1397] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.182s user 0.005s sys 0.012s
I20250111 02:57:27.103812 18624 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 076aafe3958948079e42501c6bd86153
I20250111 02:57:27.105950 18623 master_service.cc:946] ReplaceTablet: received request to replace tablet 2873325cece64db6a673d4fe08eb7409 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:27.106317 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 076aafe3958948079e42501c6bd86153 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:26 UTC) from {username='slave'} at 127.0.0.1:32868
W20250111 02:57:27.108296 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 076aafe3958948079e42501c6bd86153 because the tablet was not found. No further retry: Not found: Tablet not found: 076aafe3958948079e42501c6bd86153
I20250111 02:57:27.109797 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_23d8a352e4914f2ab823188b03644717"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:27.115304 18879 tablet_service.cc:1514] Processing DeleteTablet for tablet 076aafe3958948079e42501c6bd86153 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:26 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:27.114926 19321 raft_consensus.cc:357] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:27.115689 19321 raft_consensus.cc:383] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:27.115938 19321 raft_consensus.cc:738] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
W20250111 02:57:27.116963 18617 catalog_manager.cc:4839] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 076aafe3958948079e42501c6bd86153 because the tablet was not found. No further retry: Not found: Tablet not found: 076aafe3958948079e42501c6bd86153
I20250111 02:57:27.116685 19321 consensus_queue.cc:260] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:27.120743 18879 tablet_service.cc:1514] Processing DeleteTablet for tablet 92aa64b2de5746d1b12cc93a10c9d56e with delete_type TABLET_DATA_DELETED (replaced by tablet e46894710a72407993b36801f3b90eb0) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:27.122454 18625 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_8118351b275b46018f91df033a1270db" } modify_external_catalogs: true
I20250111 02:57:27.122704 18879 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:27.123008 18625 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_8118351b275b46018f91df033a1270db" } modify_external_catalogs: true
I20250111 02:57:27.123487 19327 tablet_replica.cc:331] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:27.124995 18879 tablet_service.cc:1514] Processing DeleteTablet for tablet 74592f80afab4513a9eb2eac351c4de5 with delete_type TABLET_DATA_DELETED (replaced by tablet f57996ef57164187b7afbff68c1b934c) from {username='slave'} at 127.0.0.1:51082
W20250111 02:57:27.126147 18617 catalog_manager.cc:4839] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 92aa64b2de5746d1b12cc93a10c9d56e because the tablet was not found. No further retry: Not found: Tablet not found: 92aa64b2de5746d1b12cc93a10c9d56e
W20250111 02:57:27.118669 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23d8a352e4914f2ab823188b03644717 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:27.130623 19696 raft_consensus.cc:491] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:27.131233 19696 raft_consensus.cc:513] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:27.132797 19330 raft_consensus.cc:357] T f778763e59a94969b3f62be87e2fd867 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:27.135365 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f778763e59a94969b3f62be87e2fd867" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:27.136682 19696 leader_election.cc:290] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:27.137179 19330 raft_consensus.cc:383] T f778763e59a94969b3f62be87e2fd867 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:27.137470 19330 raft_consensus.cc:738] T f778763e59a94969b3f62be87e2fd867 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
W20250111 02:57:27.138074 19133 leader_election.cc:343] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:27.139668 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f778763e59a94969b3f62be87e2fd867" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:27.140393 19804 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 2728215978870095747) tablet 00000000000000000000000000000000 cur holder 0x7b5800062100
I20250111 02:57:27.141136 18879 tablet_service.cc:1467] Processing CreateTablet for tablet 0c669a3ff8aa46fa8b04b86ed3bfde49 (DEFAULT_TABLE table=default.Table_fc5930652905494584fde76fff8629d8 [id=a25d2996d4d64d91980bb5a067f5b118]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.144197 19330 consensus_queue.cc:260] T f778763e59a94969b3f62be87e2fd867 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:27.153213 19177 tablet_service.cc:1467] Processing CreateTablet for tablet d83cfd85a52f438dabe49b6223d5090f (DEFAULT_TABLE table=default.Table_799d0b8328de40a7bacb58ea433fec72 [id=e9efdc6c94574ddca092dd6b9f3b8419]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.155706 19051 raft_consensus.cc:2463] T f778763e59a94969b3f62be87e2fd867 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:27.156312 19321 ts_tablet_manager.cc:1428] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.060s user 0.007s sys 0.001s
I20250111 02:57:27.154845 19177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d83cfd85a52f438dabe49b6223d5090f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.159659 19131 leader_election.cc:304] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:27.160946 19696 raft_consensus.cc:2798] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:27.161356 19696 raft_consensus.cc:491] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:27.133358 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet 076aafe3958948079e42501c6bd86153 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:26 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:27.161782 19696 raft_consensus.cc:3054] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:27.162418 19321 tablet_bootstrap.cc:492] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:27.167637 18880 tablet_service.cc:1514] Processing DeleteTablet for tablet 94b9d14959df49a6bbd80ba9e9e57ee4 with delete_type TABLET_DATA_DELETED (replaced by tablet e4bd42ad66064bcaa043746081f1c759) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:27.171131 19602 raft_consensus.cc:491] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:27.171801 19602 raft_consensus.cc:513] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:27.172577 19322 tablet_bootstrap.cc:654] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:27.177583 19602 leader_election.cc:290] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:27.178848 18879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c669a3ff8aa46fa8b04b86ed3bfde49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.179389 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74592f80afab4513a9eb2eac351c4de5" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:27.179383 19696 raft_consensus.cc:513] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:27.180370 19051 raft_consensus.cc:2388] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate d0cedab82b154d62b78809edc6f28bfa in this term.
I20250111 02:57:27.180436 18628 catalog_manager.cc:5526] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa reported cstate change: term changed from 0 to 1, leader changed from <none> to d0cedab82b154d62b78809edc6f28bfa (127.18.20.67). New cstate: current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: UNKNOWN } } }
I20250111 02:57:27.182222 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]). Expected version 2 got 0
I20250111 02:57:27.183439 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 40fdec982cf643c68ef90798f4f65ad4 (table default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927]). Expected version 1 got 0
I20250111 02:57:27.185156 19696 leader_election.cc:290] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:27.185851 19330 ts_tablet_manager.cc:1428] T f778763e59a94969b3f62be87e2fd867 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.123s user 0.003s sys 0.003s
I20250111 02:57:27.186745 19330 tablet_bootstrap.cc:492] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
W20250111 02:57:27.188937 18841 leader_election.cc:343] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Not found: Tablet not found: 74592f80afab4513a9eb2eac351c4de5
I20250111 02:57:27.189425 18841 leader_election.cc:304] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:27.192483 19602 raft_consensus.cc:3054] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:27.182754 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f778763e59a94969b3f62be87e2fd867" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:27.190966 19321 tablet_bootstrap.cc:654] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:27.197737 19133 leader_election.cc:343] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:27.198312 19326 ts_tablet_manager.cc:1918] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:27.198777 19326 log.cc:1198] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/ef70b417568049f0b496f4d8e5f7e0e4
I20250111 02:57:27.202019 18623 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 2873325cece64db6a673d4fe08eb7409
I20250111 02:57:27.203779 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 2873325cece64db6a673d4fe08eb7409 with delete_type TABLET_DATA_DELETED (replaced by tablet 3d873a0faf4a4645a3969067ae0583dc) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:27.170109 19033 tablet_service.cc:1467] Processing CreateTablet for tablet d83cfd85a52f438dabe49b6223d5090f (DEFAULT_TABLE table=default.Table_799d0b8328de40a7bacb58ea433fec72 [id=e9efdc6c94574ddca092dd6b9f3b8419]), partition=RANGE (key) PARTITION UNBOUNDED
W20250111 02:57:27.205550 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 2873325cece64db6a673d4fe08eb7409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2873325cece64db6a673d4fe08eb7409 already in progress: creating tablet
I20250111 02:57:27.206020 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d83cfd85a52f438dabe49b6223d5090f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.208156 19326 ts_tablet_manager.cc:1939] T ef70b417568049f0b496f4d8e5f7e0e4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:27.211422 19321 tablet_bootstrap.cc:492] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:27.213886 19602 raft_consensus.cc:2743] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:27.215849 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 152) took 4624 ms. Trace:
I20250111 02:57:27.216216 19326 rpcz_store.cc:276] 0111 02:57:22.591745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.591877 (+ 132us) service_pool.cc:225] Handling call
0111 02:57:26.989748 (+4397871us) ts_tablet_manager.cc:1145] Deleting tablet ef70b417568049f0b496f4d8e5f7e0e4
0111 02:57:26.989790 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:27.068092 (+ 78302us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:27.198005 (+129913us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:27.215779 (+ 17774us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4397132}
I20250111 02:57:27.213587 19168 tablet_service.cc:1467] Processing CreateTablet for tablet 0c669a3ff8aa46fa8b04b86ed3bfde49 (DEFAULT_TABLE table=default.Table_fc5930652905494584fde76fff8629d8 [id=a25d2996d4d64d91980bb5a067f5b118]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.218701 19168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c669a3ff8aa46fa8b04b86ed3bfde49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.219895 18882 tablet_service.cc:1467] Processing CreateTablet for tablet aaab428f798a44f69c80e5792d56b920 (DEFAULT_TABLE table=default.Table_e42719b273bf4b8f96fa48d1a3d6c8d8 [id=27f7d429d88b45b8b76d416b9dc582db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.220886 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaab428f798a44f69c80e5792d56b920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.223387 19326 tablet_replica.cc:331] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:27.224252 19326 raft_consensus.cc:2238] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:27.225212 19326 raft_consensus.cc:2267] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:27.231379 18628 catalog_manager.cc:5526] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa reported cstate change: term changed from 0 to 1, leader changed from <none> to d0cedab82b154d62b78809edc6f28bfa (127.18.20.67). New cstate: current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:27.233119 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet a983a670c2104398ac1a159ac3a1da7c (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19]). Expected version 4 got 0
I20250111 02:57:27.233510 19321 ts_tablet_manager.cc:1397] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.008s
I20250111 02:57:27.234805 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 2873325cece64db6a673d4fe08eb7409 with delete_type TABLET_DATA_DELETED (replaced by tablet 3d873a0faf4a4645a3969067ae0583dc) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:27.234789 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet cc7158a7fc624fa4bc816ac3721c610d (table default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605]). Expected version 2 got 0
I20250111 02:57:27.226953 19326 ts_tablet_manager.cc:1905] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:27.235491 18628 catalog_manager.cc:5526] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa reported cstate change: term changed from 0 to 1, leader changed from <none> to d0cedab82b154d62b78809edc6f28bfa (127.18.20.67). New cstate: current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } }
I20250111 02:57:27.251956 19327 raft_consensus.cc:2238] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:27.257364 19327 raft_consensus.cc:2267] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:27.261268 19696 raft_consensus.cc:491] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:27.261919 19696 raft_consensus.cc:513] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:27.265112 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf1ce65ccf54358b95665b9764fbdb0" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:27.266599 19133 leader_election.cc:343] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:27.267976 19696 leader_election.cc:290] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:27.269368 19244 heartbeater.cc:502] Master 127.18.20.126:45115 was elected leader, sending a full tablet report...
I20250111 02:57:27.274091 18879 tablet_service.cc:1467] Processing CreateTablet for tablet d3b7616fe56f4e5fb49bdde6961fa008 (DEFAULT_TABLE table=default.Table_1ab2b1f61ced4c97ad5e44f18b6e8699 [id=40eb021e024747f0a173bf87538b7281]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.275148 18879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3b7616fe56f4e5fb49bdde6961fa008. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.275579 19322 tablet_bootstrap.cc:492] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:27.276316 19327 ts_tablet_manager.cc:1905] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:27.276369 19322 ts_tablet_manager.cc:1397] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.413s user 0.008s sys 0.006s
I20250111 02:57:27.289000 19322 raft_consensus.cc:357] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:27.289994 19322 raft_consensus.cc:383] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:27.290252 19322 raft_consensus.cc:738] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:27.290864 19322 consensus_queue.cc:260] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:27.294647 19322 ts_tablet_manager.cc:1428] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250111 02:57:27.295583 19322 tablet_bootstrap.cc:492] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:27.274494 19177 tablet_service.cc:1467] Processing CreateTablet for tablet d3b7616fe56f4e5fb49bdde6961fa008 (DEFAULT_TABLE table=default.Table_1ab2b1f61ced4c97ad5e44f18b6e8699 [id=40eb021e024747f0a173bf87538b7281]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.307843 19177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3b7616fe56f4e5fb49bdde6961fa008. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.307821 18960 heartbeater.cc:346] Connected to a master server at 127.18.20.126:45115
I20250111 02:57:27.274485 19169 tablet_service.cc:1467] Processing CreateTablet for tablet aaab428f798a44f69c80e5792d56b920 (DEFAULT_TABLE table=default.Table_e42719b273bf4b8f96fa48d1a3d6c8d8 [id=27f7d429d88b45b8b76d416b9dc582db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.311750 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaab428f798a44f69c80e5792d56b920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.317018 19167 tablet_service.cc:1467] Processing CreateTablet for tablet 3514d795a0f24adcaa2accc0cf05b142 (DEFAULT_TABLE table=default.Table_4c9aebb06d554107a5448d968fdb0436 [id=3fab00618e6548de85b5d20adee4e9ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.318147 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3514d795a0f24adcaa2accc0cf05b142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.278518 19321 raft_consensus.cc:357] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:27.300541 19696 raft_consensus.cc:491] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:27.320405 19696 raft_consensus.cc:513] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:27.322247 19696 leader_election.cc:290] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:27.251669 18623 catalog_manager.cc:6596] ReplaceTablet: tablet 2873325cece64db6a673d4fe08eb7409 deleted and replaced by tablet 3d873a0faf4a4645a3969067ae0583dc
I20250111 02:57:27.323343 19321 raft_consensus.cc:383] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:27.323621 19321 raft_consensus.cc:738] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:27.324222 19321 consensus_queue.cc:260] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:27.258052 18615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.124:34593, user_credentials={real_user=slave}} blocked reactor thread for 60208.7us
I20250111 02:57:27.328245 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet ef70b417568049f0b496f4d8e5f7e0e4 (table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5]) successfully deleted
I20250111 02:57:27.332633 19696 raft_consensus.cc:491] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:27.333065 19696 raft_consensus.cc:513] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:27.335713 19696 leader_election.cc:290] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:27.344167 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_813d4e8bb0f34a46bf96ed3bee6ca86f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:27.345710 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_813d4e8bb0f34a46bf96ed3bee6ca86f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:27.347462 19321 ts_tablet_manager.cc:1428] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.114s user 0.007s sys 0.000s
I20250111 02:57:27.348696 19326 ts_tablet_manager.cc:1918] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:27.349102 19326 log.cc:1198] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/c28958a1018344a48d18e6c0f03cd0a3
I20250111 02:57:27.349851 19326 ts_tablet_manager.cc:1939] T c28958a1018344a48d18e6c0f03cd0a3 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:27.349680 19173 tablet_service.cc:1467] Processing CreateTablet for tablet 79f4500d3beb4c9bb20f84c661c3758b (DEFAULT_TABLE table=default.Table_bdacd508498241408a549fdbc6704512 [id=46a029216fdb4496bf9df1d776ccdf7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.350596 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 153) took 4757 ms. Trace:
I20250111 02:57:27.350752 19321 tablet_bootstrap.cc:492] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:27.350771 19173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79f4500d3beb4c9bb20f84c661c3758b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.363808 18890 tablet_service.cc:1467] Processing CreateTablet for tablet a6e4b222bf974a8d80b25141a5686761 (DEFAULT_TABLE table=default.Table_5e21879e67544c71af627f795fd6d940 [id=d6c0153a97cd42d49f534f48fc5b8287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.364905 18890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6e4b222bf974a8d80b25141a5686761. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.374658 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf1ce65ccf54358b95665b9764fbdb0" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:27.385205 19168 tablet_service.cc:1467] Processing CreateTablet for tablet 07f15096ea484b499fbdecec74eb6f66 (DEFAULT_TABLE table=default.Table_7653ac6289d7467590ac03b43df1b8bf [id=043fd6ad4f5c4a2685af5bbecda90328]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.385569 19024 tablet_service.cc:1467] Processing CreateTablet for tablet a6e4b222bf974a8d80b25141a5686761 (DEFAULT_TABLE table=default.Table_5e21879e67544c71af627f795fd6d940 [id=d6c0153a97cd42d49f534f48fc5b8287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.386413 19168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f15096ea484b499fbdecec74eb6f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.386555 19024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6e4b222bf974a8d80b25141a5686761. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.387848 18880 tablet_service.cc:1467] Processing CreateTablet for tablet 3514d795a0f24adcaa2accc0cf05b142 (DEFAULT_TABLE table=default.Table_4c9aebb06d554107a5448d968fdb0436 [id=3fab00618e6548de85b5d20adee4e9ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.391824 19026 tablet_service.cc:1467] Processing CreateTablet for tablet aaab428f798a44f69c80e5792d56b920 (DEFAULT_TABLE table=default.Table_e42719b273bf4b8f96fa48d1a3d6c8d8 [id=27f7d429d88b45b8b76d416b9dc582db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.393534 19028 tablet_service.cc:1467] Processing CreateTablet for tablet d3b7616fe56f4e5fb49bdde6961fa008 (DEFAULT_TABLE table=default.Table_1ab2b1f61ced4c97ad5e44f18b6e8699 [id=40eb021e024747f0a173bf87538b7281]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.395119 19029 tablet_service.cc:1467] Processing CreateTablet for tablet 0c669a3ff8aa46fa8b04b86ed3bfde49 (DEFAULT_TABLE table=default.Table_fc5930652905494584fde76fff8629d8 [id=a25d2996d4d64d91980bb5a067f5b118]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.396837 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3514d795a0f24adcaa2accc0cf05b142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.378368 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f778763e59a94969b3f62be87e2fd867" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:27.402374 19051 raft_consensus.cc:3054] T f778763e59a94969b3f62be87e2fd867 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:27.405083 19023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2873325cece64db6a673d4fe08eb7409 with delete_type TABLET_DATA_DELETED (replaced by tablet 3d873a0faf4a4645a3969067ae0583dc) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:27.407754 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b302fcba27244c193586bc25fc9305f" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:27.409482 19321 tablet_bootstrap.cc:654] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:27.350893 19326 rpcz_store.cc:276] 0111 02:57:22.593081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.593180 (+ 99us) service_pool.cc:225] Handling call
0111 02:57:27.223225 (+4630045us) ts_tablet_manager.cc:1145] Deleting tablet c28958a1018344a48d18e6c0f03cd0a3
0111 02:57:27.223268 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:27.323891 (+100623us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:27.348467 (+ 24576us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:27.350540 (+ 2073us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4629502}
I20250111 02:57:27.412884 18625 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 62c6b166a29943ceb15f093fac728c79
I20250111 02:57:27.414340 19031 tablet_service.cc:1467] Processing CreateTablet for tablet 076aafe3958948079e42501c6bd86153 (DEFAULT_TABLE table=default.Table_d234d85b2b234eef9e672be1cb4a1923 [id=dad8971929404739ae7c3b5cda2c82f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.415174 19025 tablet_service.cc:1467] Processing CreateTablet for tablet 07f15096ea484b499fbdecec74eb6f66 (DEFAULT_TABLE table=default.Table_7653ac6289d7467590ac03b43df1b8bf [id=043fd6ad4f5c4a2685af5bbecda90328]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.415988 19177 tablet_service.cc:1514] Processing DeleteTablet for tablet 62c6b166a29943ceb15f093fac728c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:27 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:27.416127 19028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3b7616fe56f4e5fb49bdde6961fa008. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.417513 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_a02982c32dcb409385f8f4e99b95423f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:27.419399 19029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c669a3ff8aa46fa8b04b86ed3bfde49. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:27.419710 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a02982c32dcb409385f8f4e99b95423f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:27.420336 19031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 076aafe3958948079e42501c6bd86153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.422013 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd364f9a6894d19826deb59a8d8a612" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:27.418525 18614 catalog_manager.cc:4852] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 62c6b166a29943ceb15f093fac728c79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62c6b166a29943ceb15f093fac728c79 already in progress: creating tablet
I20250111 02:57:27.423192 19026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaab428f798a44f69c80e5792d56b920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.423821 19025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f15096ea484b499fbdecec74eb6f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.425537 19330 tablet_bootstrap.cc:654] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:27.428362 19171 tablet_service.cc:1467] Processing CreateTablet for tablet f31ad2e1895d4a26b8e9f7cd7366ef81 (DEFAULT_TABLE table=default.Table_23d632d4476c46b78d27619201589641 [id=db5ffbbe4fe7424f943fbaca17103d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.429662 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet c28958a1018344a48d18e6c0f03cd0a3 (table default.Table_7bf5032295a34c379d00c2d9ad7e417f [id=f6f9076d18654f339a6feac0c468e265]) successfully deleted
I20250111 02:57:27.430042 19171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31ad2e1895d4a26b8e9f7cd7366ef81. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:27.432957 19131 leader_election.cc:343] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:27.433609 19131 leader_election.cc:343] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:27.433962 19131 leader_election.cc:304] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:27.433728 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 154) took 4649 ms. Trace:
W20250111 02:57:27.434998 19131 leader_election.cc:343] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:27.435379 18629 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]). Expected version 2 got 0
I20250111 02:57:27.435670 19326 rpcz_store.cc:276] 0111 02:57:22.784468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:22.784615 (+ 147us) service_pool.cc:225] Handling call
0111 02:57:27.413517 (+4628902us) ts_tablet_manager.cc:1145] Deleting tablet 3052e9707de14e1f9e38236d56908ed3
0111 02:57:27.413565 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:27.433663 (+ 20098us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4628305}
I20250111 02:57:27.437325 19034 tablet_service.cc:1514] Processing DeleteTablet for tablet 62c6b166a29943ceb15f093fac728c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:27 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:27.439855 19027 tablet_service.cc:1467] Processing CreateTablet for tablet f31ad2e1895d4a26b8e9f7cd7366ef81 (DEFAULT_TABLE table=default.Table_23d632d4476c46b78d27619201589641 [id=db5ffbbe4fe7424f943fbaca17103d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.441099 19027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31ad2e1895d4a26b8e9f7cd7366ef81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.442533 18629 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet cc7158a7fc624fa4bc816ac3721c610d (table default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605]). Expected version 2 got 0
I20250111 02:57:27.443575 18629 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet a983a670c2104398ac1a159ac3a1da7c (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19]). Expected version 4 got 0
I20250111 02:57:27.456930 19174 tablet_service.cc:1467] Processing CreateTablet for tablet a6e4b222bf974a8d80b25141a5686761 (DEFAULT_TABLE table=default.Table_5e21879e67544c71af627f795fd6d940 [id=d6c0153a97cd42d49f534f48fc5b8287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.458002 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6e4b222bf974a8d80b25141a5686761. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:27.458096 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 3052e9707de14e1f9e38236d56908ed3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3052e9707de14e1f9e38236d56908ed3 already in progress: creating tablet
I20250111 02:57:27.456000 19696 raft_consensus.cc:2743] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:27.461225 19327 ts_tablet_manager.cc:1918] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:27.461580 19327 log.cc:1198] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/5fe8aa967fcb4bd7a48f6f0f69bea496
I20250111 02:57:27.462260 19327 ts_tablet_manager.cc:1939] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:27.462015 19165 tablet_service.cc:1467] Processing CreateTablet for tablet 56fb0bf949904208af003f08caae7594 (DEFAULT_TABLE table=default.Table_23a3f1c40c15458e93c2ef79a3c9e9e2 [id=c12add3740174f66996466994ebe9971]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.463318 19165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56fb0bf949904208af003f08caae7594. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.465071 19327 tablet_replica.cc:331] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:27.465760 19327 raft_consensus.cc:2238] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:27.466177 19327 raft_consensus.cc:2267] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:27.468166 19170 tablet_service.cc:1467] Processing CreateTablet for tablet 3c57d85390324bbc9ea30a709faec01c (DEFAULT_TABLE table=default.Table_52d947397a774f96a4fdb5087da4e19a [id=7196ac9c0c4d4dada6c94803e1220caa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.469566 19170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c57d85390324bbc9ea30a709faec01c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.469827 19033 tablet_service.cc:1467] Processing CreateTablet for tablet 79f4500d3beb4c9bb20f84c661c3758b (DEFAULT_TABLE table=default.Table_bdacd508498241408a549fdbc6704512 [id=46a029216fdb4496bf9df1d776ccdf7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.474298 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79f4500d3beb4c9bb20f84c661c3758b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.484267 19101 heartbeater.cc:346] Connected to a master server at 127.18.20.126:45115
I20250111 02:57:27.485618 19177 tablet_service.cc:1467] Processing CreateTablet for tablet 076aafe3958948079e42501c6bd86153 (DEFAULT_TABLE table=default.Table_d234d85b2b234eef9e672be1cb4a1923 [id=dad8971929404739ae7c3b5cda2c82f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.487051 19177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 076aafe3958948079e42501c6bd86153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.489684 18629 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 40fdec982cf643c68ef90798f4f65ad4 (table default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927]). Expected version 1 got 0
I20250111 02:57:27.490545 18629 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 7ea78feee64340e39404cba0f1bf04db (table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5]). Expected version 1 got 0
I20250111 02:57:27.436892 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 156) took 4250 ms. Trace:
I20250111 02:57:27.504980 19326 rpcz_store.cc:276] 0111 02:57:23.186687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:23.186869 (+ 182us) service_pool.cc:225] Handling call
0111 02:57:27.436534 (+4249665us) ts_tablet_manager.cc:1145] Deleting tablet b26e5db463e94d919640a07a2f8574a8
0111 02:57:27.436578 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:27.436836 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4248965}
I20250111 02:57:27.506556 18629 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 845da3ee0863430ea3fe9de8c84f2c17 (table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3]). Expected version 2 got 0
I20250111 02:57:27.512557 19169 tablet_service.cc:1467] Processing CreateTablet for tablet 48dd0bccf6854d37a7ef95b5ab06308e (DEFAULT_TABLE table=default.Table_60c8bd1736a84a2d9fd52866d624704e [id=486234c549894967b37acb3ea53d3b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.513015 19244 heartbeater.cc:502] Master 127.18.20.126:45115 was elected leader, sending a full tablet report...
I20250111 02:57:27.517541 19327 ts_tablet_manager.cc:1905] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:27.392427 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd364f9a6894d19826deb59a8d8a612" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:27.520864 18879 tablet_service.cc:1467] Processing CreateTablet for tablet 076aafe3958948079e42501c6bd86153 (DEFAULT_TABLE table=default.Table_d234d85b2b234eef9e672be1cb4a1923 [id=dad8971929404739ae7c3b5cda2c82f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.522226 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet 62c6b166a29943ceb15f093fac728c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:27 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:27.513679 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48dd0bccf6854d37a7ef95b5ab06308e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.522864 18623 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_53adc14208674563b5bda158cf0c68a3" } new_table_name: "default.Table_5e7c139097914dd7b7c8e55058d5290d" modify_external_catalogs: true
I20250111 02:57:27.537865 18621 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]). Expected version 2 got 0
I20250111 02:57:27.538604 18621 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet 40fdec982cf643c68ef90798f4f65ad4 (table default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927]). Expected version 1 got 0
I20250111 02:57:27.539503 18621 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet a983a670c2104398ac1a159ac3a1da7c (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19]). Expected version 4 got 0
I20250111 02:57:27.540719 18621 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet cc7158a7fc624fa4bc816ac3721c610d (table default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605]). Expected version 2 got 0
I20250111 02:57:27.541740 18621 catalog_manager.cc:5526] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:27.549981 18879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 076aafe3958948079e42501c6bd86153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.580816 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_98958a28faeb43e8a3b34fcf20d936aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:27.582600 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98958a28faeb43e8a3b34fcf20d936aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:27.590629 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet b26e5db463e94d919640a07a2f8574a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b26e5db463e94d919640a07a2f8574a8 already in progress: creating tablet
I20250111 02:57:27.606833 18625 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_7653ac6289d7467590ac03b43df1b8bf" } modify_external_catalogs: true
I20250111 02:57:27.607589 18625 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_7653ac6289d7467590ac03b43df1b8bf" } modify_external_catalogs: true
W20250111 02:57:27.610395 18623 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 220da826a0f140d59193e8ad72096b60 consensus state has no leader
I20250111 02:57:27.613950 18623 master_service.cc:946] ReplaceTablet: received request to replace tablet d6da575f052d4b20a8e41f0bd9dab7e4 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:27.624759 18624 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_baa2b0022e914aa5b485db90037be4af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:27.626792 18624 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_baa2b0022e914aa5b485db90037be4af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:27.628782 19023 tablet_service.cc:1467] Processing CreateTablet for tablet 3514d795a0f24adcaa2accc0cf05b142 (DEFAULT_TABLE table=default.Table_4c9aebb06d554107a5448d968fdb0436 [id=3fab00618e6548de85b5d20adee4e9ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.629788 19023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3514d795a0f24adcaa2accc0cf05b142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.632339 19032 tablet_service.cc:1467] Processing CreateTablet for tablet 48dd0bccf6854d37a7ef95b5ab06308e (DEFAULT_TABLE table=default.Table_60c8bd1736a84a2d9fd52866d624704e [id=486234c549894967b37acb3ea53d3b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.633934 19032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48dd0bccf6854d37a7ef95b5ab06308e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.636317 19321 tablet_bootstrap.cc:492] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:27.639379 19321 ts_tablet_manager.cc:1397] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.289s user 0.009s sys 0.003s
I20250111 02:57:27.639999 19034 tablet_service.cc:1467] Processing CreateTablet for tablet 56fb0bf949904208af003f08caae7594 (DEFAULT_TABLE table=default.Table_23a3f1c40c15458e93c2ef79a3c9e9e2 [id=c12add3740174f66996466994ebe9971]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.641479 19034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56fb0bf949904208af003f08caae7594. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.658226 18882 tablet_service.cc:1467] Processing CreateTablet for tablet 56fb0bf949904208af003f08caae7594 (DEFAULT_TABLE table=default.Table_23a3f1c40c15458e93c2ef79a3c9e9e2 [id=c12add3740174f66996466994ebe9971]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.659346 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56fb0bf949904208af003f08caae7594. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.663374 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 3c57d85390324bbc9ea30a709faec01c (DEFAULT_TABLE table=default.Table_52d947397a774f96a4fdb5087da4e19a [id=7196ac9c0c4d4dada6c94803e1220caa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.664578 18878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c57d85390324bbc9ea30a709faec01c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.666785 18885 tablet_service.cc:1467] Processing CreateTablet for tablet 48dd0bccf6854d37a7ef95b5ab06308e (DEFAULT_TABLE table=default.Table_60c8bd1736a84a2d9fd52866d624704e [id=486234c549894967b37acb3ea53d3b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.668335 18885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48dd0bccf6854d37a7ef95b5ab06308e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.668707 19321 raft_consensus.cc:357] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:27.669385 19321 raft_consensus.cc:383] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:27.669631 19321 raft_consensus.cc:738] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:27.671098 18887 tablet_service.cc:1514] Processing DeleteTablet for tablet 199ca8692f39445f93d8c27e2134a32c with delete_type TABLET_DATA_DELETED (replaced by tablet 29f27ed2c53d43d7a51ee2e45afaf9fc) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:27.672549 18881 tablet_service.cc:1467] Processing CreateTablet for tablet 79f4500d3beb4c9bb20f84c661c3758b (DEFAULT_TABLE table=default.Table_bdacd508498241408a549fdbc6704512 [id=46a029216fdb4496bf9df1d776ccdf7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.673931 18881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79f4500d3beb4c9bb20f84c661c3758b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.677762 18886 tablet_service.cc:1467] Processing CreateTablet for tablet 07f15096ea484b499fbdecec74eb6f66 (DEFAULT_TABLE table=default.Table_7653ac6289d7467590ac03b43df1b8bf [id=043fd6ad4f5c4a2685af5bbecda90328]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.679641 18884 tablet_service.cc:1467] Processing CreateTablet for tablet f31ad2e1895d4a26b8e9f7cd7366ef81 (DEFAULT_TABLE table=default.Table_23d632d4476c46b78d27619201589641 [id=db5ffbbe4fe7424f943fbaca17103d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.559326 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b302fcba27244c193586bc25fc9305f" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:27.687151 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 157) took 4094 ms. Trace:
I20250111 02:57:27.687505 19326 rpcz_store.cc:276] 0111 02:57:23.593157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:23.593298 (+ 141us) service_pool.cc:225] Handling call
0111 02:57:27.506322 (+3913024us) ts_tablet_manager.cc:1145] Deleting tablet 0dbb63ab78f84995ae6d9fb2aae78dcf
0111 02:57:27.506381 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:27.687044 (+180663us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3912327}
W20250111 02:57:27.689016 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 0dbb63ab78f84995ae6d9fb2aae78dcf because the tablet was not found. No further retry: Not found: Tablet not found: 0dbb63ab78f84995ae6d9fb2aae78dcf
I20250111 02:57:27.690135 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 164) took 3330 ms. Trace:
I20250111 02:57:27.690436 19326 rpcz_store.cc:276] 0111 02:57:24.359894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:24.363167 (+ 3273us) service_pool.cc:225] Handling call
0111 02:57:27.689808 (+3326641us) ts_tablet_manager.cc:1145] Deleting tablet 0ef86e1fd412484db83e7d39c9059a90
0111 02:57:27.689852 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:27.690077 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3326005}
W20250111 02:57:27.691819 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 0ef86e1fd412484db83e7d39c9059a90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ef86e1fd412484db83e7d39c9059a90 already in progress: creating tablet
I20250111 02:57:27.691898 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 (table default.Table_f731216a607f4fdaadd2e8dbedf7552d [id=e77c4aa40ab44fb3bc98193e503be4cf]) successfully deleted
I20250111 02:57:27.692721 19326 tablet_replica.cc:331] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:27.694010 19326 raft_consensus.cc:2238] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:27.694624 19326 raft_consensus.cc:2267] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:27.696911 19326 ts_tablet_manager.cc:1905] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:27.699998 19030 tablet_service.cc:1467] Processing CreateTablet for tablet 3c57d85390324bbc9ea30a709faec01c (DEFAULT_TABLE table=default.Table_52d947397a774f96a4fdb5087da4e19a [id=7196ac9c0c4d4dada6c94803e1220caa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:27.701625 19030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c57d85390324bbc9ea30a709faec01c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.702267 18886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f15096ea484b499fbdecec74eb6f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:27.704005 18884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31ad2e1895d4a26b8e9f7cd7366ef81. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:27.708846 18614 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 845da3ee0863430ea3fe9de8c84f2c17 (table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 845da3ee0863430ea3fe9de8c84f2c17 consensus state has no leader
I20250111 02:57:27.708448 18906 raft_consensus.cc:3054] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:27.690969 19133 leader_election.cc:343] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:27.727566 19133 leader_election.cc:304] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:27.728710 19860 raft_consensus.cc:2743] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:27.730188 19133 leader_election.cc:343] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:27.730831 19133 leader_election.cc:304] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:27.731856 19860 raft_consensus.cc:2743] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:27.722883 19863 log.cc:926] Time spent T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Append to log took a long time: real 0.145s user 0.020s sys 0.003s
I20250111 02:57:27.689603 19321 consensus_queue.cc:260] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:27.737236 19321 ts_tablet_manager.cc:1428] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.097s user 0.004s sys 0.002s
I20250111 02:57:27.738253 19322 tablet_bootstrap.cc:654] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:27.772292 19051 raft_consensus.cc:2463] T f778763e59a94969b3f62be87e2fd867 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
W20250111 02:57:27.773444 18989 consensus_peers.cc:487] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:27.775331 19131 leader_election.cc:304] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:27.779862 19321 tablet_bootstrap.cc:492] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:27.786475 19754 raft_consensus.cc:2798] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:27.787282 19754 raft_consensus.cc:695] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:27.788057 19754 consensus_queue.cc:237] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:27.799441 19321 tablet_bootstrap.cc:654] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:27.802901 19244 heartbeater.cc:346] Connected to a master server at 127.18.20.126:45115
I20250111 02:57:27.805308 19244 heartbeater.cc:502] Master 127.18.20.126:45115 was elected leader, sending a full tablet report...
I20250111 02:57:27.830569 19321 tablet_bootstrap.cc:492] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:27.831149 19321 ts_tablet_manager.cc:1397] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.000s
I20250111 02:57:27.833479 19321 raft_consensus.cc:357] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:27.841588 19321 raft_consensus.cc:383] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:27.843173 19321 raft_consensus.cc:738] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:27.844055 19321 consensus_queue.cc:260] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:27.848088 19321 ts_tablet_manager.cc:1428] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20250111 02:57:27.850953 19321 tablet_bootstrap.cc:492] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:27.848528 19874 tablet.cc:1721] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250111 02:57:27.857004 18630 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47522 (request call id 231) took 2797 ms. Trace:
I20250111 02:57:27.861577 19032 tablet_service.cc:1514] Processing DeleteTablet for tablet 0267a92995ea460786dc26813b5895cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:22 UTC) from {username='slave'} at 127.0.0.1:36190
W20250111 02:57:27.862015 18627 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.66:37049 (request call id 37) took 3094 ms (client timeout 1000 ms). Trace:
I20250111 02:57:27.862432 19032 tablet_service.cc:1514] Processing DeleteTablet for tablet b54311f4269d4575b8998d7f545000e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 845da3ee0863430ea3fe9de8c84f2c17) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:27.862928 19022 tablet_service.cc:1514] Processing DeleteTablet for tablet d8128b3941d54ce2a72bf2ae762a774f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:36190
W20250111 02:57:27.862337 18627 rpcz_store.cc:269] 0111 02:57:24.768011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:24.848308 (+ 80297us) service_pool.cc:225] Handling call
0111 02:57:27.861936 (+3013628us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":7}
I20250111 02:57:27.868067 19022 tablet_service.cc:1514] Processing DeleteTablet for tablet c28958a1018344a48d18e6c0f03cd0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:17 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:27.868705 18627 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_5bc6e85b9ba443aa8ddc113f6b5debbc" } modify_external_catalogs: true
I20250111 02:57:27.869256 18627 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_5bc6e85b9ba443aa8ddc113f6b5debbc" } modify_external_catalogs: true
I20250111 02:57:27.869948 19321 tablet_bootstrap.cc:654] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:27.857371 18630 rpcz_store.cc:276] 0111 02:57:25.059277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:25.059444 (+ 167us) service_pool.cc:225] Handling call
0111 02:57:25.217053 (+157609us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d53986 00007fd427ac0644 00007fd427ac05d2 00007fd427ac0501 00007fd427ac047c 00007fd427abf732 00007fd425d79dfc 00007fd425d79236
0111 02:57:25.217808 (+ 755us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0111 02:57:25.218451 (+ 643us) catalog_manager.cc:3627] Apply alter schema
0111 02:57:25.225590 (+ 7139us) catalog_manager.cc:3659] Acquired catalog manager lock
0111 02:57:25.225873 (+ 283us) catalog_manager.cc:3817] Updating metadata on disk
0111 02:57:25.361623 (+135750us) catalog_manager.cc:3848] Committing alterations to in-memory state
0111 02:57:27.856942 (+2495319us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:25.345552 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:25.346006 (+ 454us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:25.346041 (+ 35us) write_op.cc:435] Acquired schema lock
0111 02:57:25.346061 (+ 20us) tablet.cc:659] Decoding operations
0111 02:57:25.346382 (+ 321us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:25.346433 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:25.346462 (+ 29us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:25.346669 (+ 207us) tablet.cc:698] Row locks acquired
0111 02:57:25.346681 (+ 12us) write_op.cc:260] PREPARE: finished
0111 02:57:25.346788 (+ 107us) write_op.cc:270] Start()
0111 02:57:25.346872 (+ 84us) write_op.cc:276] Timestamp: P: 1736564245346769 usec, L: 0
0111 02:57:25.346887 (+ 15us) op_driver.cc:348] REPLICATION: starting
0111 02:57:25.347477 (+ 590us) log.cc:844] Serialized 405 byte log entry
0111 02:57:25.356190 (+ 8713us) op_driver.cc:464] REPLICATION: finished
0111 02:57:25.358884 (+ 2694us) write_op.cc:301] APPLY: starting
0111 02:57:25.358952 (+ 68us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:25.359753 (+ 801us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:25.359915 (+ 162us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:25.360162 (+ 247us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:25.360571 (+ 409us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:25.360646 (+ 75us) write_op.cc:312] APPLY: finished
0111 02:57:25.360879 (+ 233us) log.cc:844] Serialized 23 byte log entry
0111 02:57:25.361057 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:25.361143 (+ 86us) write_op.cc:454] Released schema lock
0111 02:57:25.361182 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:25.396368 (+ 35186us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:25.396396 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":357376896,"child_traces":[["op",{"apply.queue_time_us":2516,"apply.run_cpu_time_us":1816,"apply.run_wall_time_us":2555,"num_ops":1,"prepare.queue_time_us":118419,"prepare.run_cpu_time_us":2289,"prepare.run_wall_time_us":50817,"raft.queue_time_us":81,"raft.run_cpu_time_us":585,"raft.run_wall_time_us":634,"replication_time_us":9193}]]}
I20250111 02:57:27.874781 18630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_a7b2b5af5dfb439a84159b75de0d03b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:27.876523 18630 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7b2b5af5dfb439a84159b75de0d03b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:27.881856 18626 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.66:37049 (request call id 38) took 2057 ms (client timeout 1000 ms). Trace:
W20250111 02:57:27.882309 18626 rpcz_store.cc:269] 0111 02:57:25.823978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:25.841258 (+ 17280us) service_pool.cc:225] Handling call
0111 02:57:27.881715 (+2040457us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2892,"reported_tablets":16}
I20250111 02:57:27.883684 18626 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_e091dc5f47d9492aa86ff883e04fe415"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:27.888144 19025 tablet_service.cc:1514] Processing DeleteTablet for tablet ef70b417568049f0b496f4d8e5f7e0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 7ea78feee64340e39404cba0f1bf04db) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:27.888648 19022 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe4e60db4244a9a9696153b630c575b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:27.889544 19022 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2377893bfc426fa00d5ddacd9c5961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:21 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:27.889519 19024 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:36190
W20250111 02:57:27.892191 18626 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e091dc5f47d9492aa86ff883e04fe415 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:27.904371 19133 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 53163.7us
W20250111 02:57:27.915104 18622 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.67:53171 (request call id 38) took 3091 ms (client timeout 1000 ms). Trace:
W20250111 02:57:27.915627 18622 rpcz_store.cc:269] 0111 02:57:24.819434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:24.898185 (+ 78751us) service_pool.cc:225] Handling call
0111 02:57:27.910991 (+3012806us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43350,"reported_tablets":10}
I20250111 02:57:27.917896 18622 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_23d8a352e4914f2ab823188b03644717" } new_table_name: "default.Table_8eae98e1e49c4f6c9e52644256812eaa" modify_external_catalogs: true
W20250111 02:57:27.924033 18628 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.67:53171 (request call id 39) took 2044 ms (client timeout 1000 ms). Trace:
W20250111 02:57:27.924507 18628 rpcz_store.cc:269] 0111 02:57:25.879507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:25.956458 (+ 76951us) service_pool.cc:225] Handling call
0111 02:57:27.923965 (+1967507us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:27.330393 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:27.330768 (+ 375us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:27.330793 (+ 25us) write_op.cc:435] Acquired schema lock
0111 02:57:27.330806 (+ 13us) tablet.cc:659] Decoding operations
0111 02:57:27.331271 (+ 465us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:27.331306 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:27.331328 (+ 22us) tablet.cc:682] Acquiring locks for 3 operations
0111 02:57:27.331566 (+ 238us) tablet.cc:698] Row locks acquired
0111 02:57:27.331579 (+ 13us) write_op.cc:260] PREPARE: finished
0111 02:57:27.331693 (+ 114us) write_op.cc:270] Start()
0111 02:57:27.331776 (+ 83us) write_op.cc:276] Timestamp: P: 1736564247331660 usec, L: 0
0111 02:57:27.331790 (+ 14us) op_driver.cc:348] REPLICATION: starting
0111 02:57:27.332284 (+ 494us) log.cc:844] Serialized 1284 byte log entry
0111 02:57:27.409104 (+ 76820us) op_driver.cc:464] REPLICATION: finished
0111 02:57:27.410993 (+ 1889us) write_op.cc:301] APPLY: starting
0111 02:57:27.411045 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:27.411189 (+ 144us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:27.411279 (+ 90us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:27.411582 (+ 303us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:27.411726 (+ 144us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0111 02:57:27.411764 (+ 38us) write_op.cc:312] APPLY: finished
0111 02:57:27.411945 (+ 181us) log.cc:844] Serialized 35 byte log entry
0111 02:57:27.412059 (+ 114us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:27.412123 (+ 64us) write_op.cc:454] Released schema lock
0111 02:57:27.412157 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":115201,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":1774,"apply.run_cpu_time_us":1352,"apply.run_wall_time_us":1352,"num_ops":3,"prepare.queue_time_us":4688,"prepare.run_cpu_time_us":2087,"prepare.run_wall_time_us":2089,"replication_time_us":77243}]]}
I20250111 02:57:27.929229 19849 consensus_queue.cc:1035] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:27.929628 18628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_e2a51ae779294deb90786269e7a8f13d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:27.934705 18628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2a51ae779294deb90786269e7a8f13d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:27.938175 19875 tablet.cc:1721] T 40fdec982cf643c68ef90798f4f65ad4 P 54628fbcec2f4359aa9e26a734d4d29f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250111 02:57:27.944721 19330 tablet_bootstrap.cc:492] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:27.954288 19330 ts_tablet_manager.cc:1397] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.768s user 0.015s sys 0.011s
I20250111 02:57:27.922894 19169 tablet_service.cc:1514] Processing DeleteTablet for tablet 0267a92995ea460786dc26813b5895cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:22 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:27.923202 19167 tablet_service.cc:1514] Processing DeleteTablet for tablet 32afc2f158254811836fcf00f7f409ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:19 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:27.923302 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet 74592f80afab4513a9eb2eac351c4de5 with delete_type TABLET_DATA_DELETED (replaced by tablet f57996ef57164187b7afbff68c1b934c) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:27.923741 19177 tablet_service.cc:1514] Processing DeleteTablet for tablet 199ca8692f39445f93d8c27e2134a32c with delete_type TABLET_DATA_DELETED (replaced by tablet 29f27ed2c53d43d7a51ee2e45afaf9fc) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:27.924219 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet 94b9d14959df49a6bbd80ba9e9e57ee4 with delete_type TABLET_DATA_DELETED (replaced by tablet e4bd42ad66064bcaa043746081f1c759) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:27.924901 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 92aa64b2de5746d1b12cc93a10c9d56e with delete_type TABLET_DATA_DELETED (replaced by tablet e46894710a72407993b36801f3b90eb0) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:27.925446 19165 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2377893bfc426fa00d5ddacd9c5961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:21 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:27.926645 19176 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:27.988373 19330 raft_consensus.cc:357] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:28.008210 19883 tablet_replica.cc:331] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:27.990768 19330 raft_consensus.cc:383] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250111 02:57:28.021003 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 46838.9us
I20250111 02:57:28.015996 19330 raft_consensus.cc:738] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:28.046495 19874 tablet.cc:1721] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250111 02:57:28.047857 19330 consensus_queue.cc:260] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:28.061811 19849 consensus_queue.cc:1035] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:28.084159 19321 tablet_bootstrap.cc:492] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:28.085148 19321 ts_tablet_manager.cc:1397] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.234s user 0.004s sys 0.016s
I20250111 02:57:28.089253 19860 raft_consensus.cc:491] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:28.095293 19330 ts_tablet_manager.cc:1428] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.135s user 0.013s sys 0.002s
I20250111 02:57:28.090608 19860 raft_consensus.cc:513] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:28.097566 19330 tablet_bootstrap.cc:492] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:28.099538 19860 leader_election.cc:290] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:28.100610 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a03880a8b0e46b7858ed1c2c2dddcaa" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:28.103371 19131 leader_election.cc:343] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:28.090468 19886 tablet.cc:1721] T 9192160e931445f1acd207604192f759 P 54628fbcec2f4359aa9e26a734d4d29f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250111 02:57:28.106760 19173 consensus_meta.cc:287] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa: Time spent flushing consensus metadata: real 0.564s user 0.010s sys 0.001s
I20250111 02:57:28.131858 19326 ts_tablet_manager.cc:1918] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250111 02:57:28.138384 19326 log.cc:1198] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/b8e8c72c99ac4c369957b571d74baffe
I20250111 02:57:28.141027 19326 ts_tablet_manager.cc:1939] T b8e8c72c99ac4c369957b571d74baffe P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:28.151314 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 165) took 3674 ms. Trace:
I20250111 02:57:28.153405 19889 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:28.149053 19330 tablet_bootstrap.cc:654] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:28.154294 19326 rpcz_store.cc:276] 0111 02:57:24.476857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:24.477000 (+ 143us) service_pool.cc:225] Handling call
0111 02:57:27.692517 (+3215517us) ts_tablet_manager.cc:1145] Deleting tablet b8e8c72c99ac4c369957b571d74baffe
0111 02:57:27.692574 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:28.011797 (+319223us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:28.131313 (+119516us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:28.151166 (+ 19853us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3214734}
I20250111 02:57:28.168447 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet b8e8c72c99ac4c369957b571d74baffe (table default.Table_35d7f864912b40bcb5251a90d40c7849 [id=772d994bee2d4e34bd48912acecf8d48]) successfully deleted
I20250111 02:57:28.173182 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 166) took 3422 ms. Trace:
I20250111 02:57:28.176008 19326 rpcz_store.cc:276] 0111 02:57:24.750323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:24.750508 (+ 185us) service_pool.cc:225] Handling call
0111 02:57:28.172631 (+3422123us) ts_tablet_manager.cc:1145] Deleting tablet 5118d4b9cfbb49e8af6af56180858336
0111 02:57:28.172694 (+ 63us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:28.173069 (+ 375us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3418592}
W20250111 02:57:28.178264 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 5118d4b9cfbb49e8af6af56180858336 because the tablet was not found. No further retry: Not found: Tablet not found: 5118d4b9cfbb49e8af6af56180858336
I20250111 02:57:28.179949 19326 tablet_replica.cc:331] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:28.203697 19330 tablet_bootstrap.cc:492] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:28.206147 19839 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:28.206503 19330 ts_tablet_manager.cc:1397] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.109s user 0.016s sys 0.009s
I20250111 02:57:28.207342 19839 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:28.217831 19886 tablet.cc:1721] T d8ef3f45e5784a29b3b32cc0de6bac84 P 54628fbcec2f4359aa9e26a734d4d29f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250111 02:57:28.222707 19683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40212 (request call id 0) took 2323 ms. Trace:
I20250111 02:57:28.196700 19860 consensus_queue.cc:1035] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:28.156981 19321 raft_consensus.cc:357] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:28.227152 19321 raft_consensus.cc:383] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:28.227598 19321 raft_consensus.cc:738] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:28.211248 19874 tablet.cc:1721] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250111 02:57:28.228329 19321 consensus_queue.cc:260] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:28.212452 19330 raft_consensus.cc:357] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:28.243896 19330 raft_consensus.cc:383] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:28.245095 19330 raft_consensus.cc:738] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:28.247664 19330 consensus_queue.cc:260] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:28.250105 19321 ts_tablet_manager.cc:1428] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.157s user 0.009s sys 0.005s
I20250111 02:57:28.252363 19321 tablet_bootstrap.cc:492] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:28.263218 19330 ts_tablet_manager.cc:1428] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.056s user 0.011s sys 0.005s
I20250111 02:57:28.265197 19330 tablet_bootstrap.cc:492] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:28.228166 19864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:28.265918 19864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:28.226697 19805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8784b9603f8544b79373158aace063af. Latest consensus state: current_term: 2 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:28.154769 19889 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:28.271896 19805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:28.278200 19883 raft_consensus.cc:2238] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:28.279846 19883 raft_consensus.cc:2267] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:28.285818 19849 raft_consensus.cc:491] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:28.287760 19849 raft_consensus.cc:513] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:28.309631 19326 raft_consensus.cc:2238] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:28.297982 19849 leader_election.cc:290] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:28.310657 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea78feee64340e39404cba0f1bf04db" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:28.313752 19192 raft_consensus.cc:2463] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:28.320406 18986 leader_election.cc:304] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:28.327551 19326 raft_consensus.cc:2267] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:28.332152 19885 raft_consensus.cc:491] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:28.332558 19481 raft_consensus.cc:2798] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:28.334090 19481 raft_consensus.cc:491] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:28.334584 19481 raft_consensus.cc:3054] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:28.334090 19885 raft_consensus.cc:513] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:28.338353 19326 ts_tablet_manager.cc:1905] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:28.352661 19885 leader_election.cc:290] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:28.355484 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b302fcba27244c193586bc25fc9305f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:28.358506 19192 raft_consensus.cc:2463] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:28.365923 19861 raft_consensus.cc:491] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:28.367439 19861 raft_consensus.cc:513] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:28.364163 18986 leader_election.cc:304] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:28.374210 19885 raft_consensus.cc:2798] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:28.375958 19885 raft_consensus.cc:491] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:28.377849 19885 raft_consensus.cc:3054] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:28.373773 19839 leader_election.cc:290] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8784b9603f8544b79373158aace063af (127.18.20.125:40359), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:28.310828 19896 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250111 02:57:28.399433 19330 tablet_bootstrap.cc:654] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:28.422240 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 266 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:28.424647 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8414653def014ce98bc38d87aac75bae" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:28.448379 18882 consensus_meta.cc:287] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent flushing consensus metadata: real 0.654s user 0.017s sys 0.000s
I20250111 02:57:28.456805 19860 raft_consensus.cc:491] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:28.461810 19860 raft_consensus.cc:513] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:28.450331 19883 ts_tablet_manager.cc:1905] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:28.491550 19861 leader_election.cc:290] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:28.501927 18879 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51082 (request call id 188) took 1029 ms. Trace:
I20250111 02:57:28.520537 19889 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
W20250111 02:57:28.530560 19131 leader_election.cc:343] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:28.547518 19908 raft_consensus.cc:491] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:28.549901 19860 leader_election.cc:290] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:28.552453 19330 tablet_bootstrap.cc:492] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:28.560104 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0158a8849bf44a4fbb5daca22883478f" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:28.583878 18624 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47522 (request call id 257) took 1017 ms. Trace:
I20250111 02:57:28.585078 18624 rpcz_store.cc:276] 0111 02:57:27.566850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.624039 (+ 57189us) service_pool.cc:225] Handling call
0111 02:57:27.627356 (+ 3317us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 02:57:27.629295 (+ 1939us) catalog_manager.cc:2122] Created new table and tablet info
0111 02:57:28.581970 (+952675us) catalog_manager.cc:2182] Wrote table and tablets to system table
0111 02:57:28.583034 (+ 1064us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0111 02:57:28.583714 (+ 680us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:28.120413 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:28.121305 (+ 892us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:28.121329 (+ 24us) write_op.cc:435] Acquired schema lock
0111 02:57:28.121343 (+ 14us) tablet.cc:659] Decoding operations
0111 02:57:28.121688 (+ 345us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:28.121828 (+ 140us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:28.121854 (+ 26us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:28.122335 (+ 481us) tablet.cc:698] Row locks acquired
0111 02:57:28.122372 (+ 37us) write_op.cc:260] PREPARE: finished
0111 02:57:28.122555 (+ 183us) write_op.cc:270] Start()
0111 02:57:28.122761 (+ 206us) write_op.cc:276] Timestamp: P: 1736564248122464 usec, L: 0
0111 02:57:28.122840 (+ 79us) op_driver.cc:348] REPLICATION: starting
0111 02:57:28.124729 (+ 1889us) log.cc:844] Serialized 433 byte log entry
0111 02:57:28.572252 (+447523us) op_driver.cc:464] REPLICATION: finished
0111 02:57:28.575433 (+ 3181us) write_op.cc:301] APPLY: starting
0111 02:57:28.575685 (+ 252us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:28.576055 (+ 370us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:28.576533 (+ 478us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:28.578716 (+ 2183us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:28.579387 (+ 671us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0111 02:57:28.579551 (+ 164us) write_op.cc:312] APPLY: finished
0111 02:57:28.580164 (+ 613us) log.cc:844] Serialized 29 byte log entry
0111 02:57:28.580707 (+ 543us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:28.581001 (+ 294us) write_op.cc:454] Released schema lock
0111 02:57:28.581103 (+ 102us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:28.582242 (+ 1139us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:28.582354 (+ 112us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2461,"apply.run_cpu_time_us":6682,"apply.run_wall_time_us":6676,"num_ops":2,"prepare.queue_time_us":489377,"prepare.run_cpu_time_us":10053,"prepare.run_wall_time_us":40013,"raft.queue_time_us":30306,"raft.run_cpu_time_us":3161,"raft.run_wall_time_us":161070,"replication_time_us":448935,"spinlock_wait_cycles":112512,"thread_start_us":34108,"threads_started":2}]]}
I20250111 02:57:28.553586 19330 ts_tablet_manager.cc:1397] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.289s user 0.026s sys 0.012s
I20250111 02:57:28.591508 19321 tablet_bootstrap.cc:654] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:28.595726 19860 raft_consensus.cc:491] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250111 02:57:28.615872 19131 leader_election.cc:343] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:28.626109 19860 raft_consensus.cc:513] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:28.636406 19481 raft_consensus.cc:513] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:28.618468 19330 raft_consensus.cc:357] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:28.655459 19330 raft_consensus.cc:383] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:28.600453 19885 raft_consensus.cc:513] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:28.663020 18989 consensus_peers.cc:487] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:28.671217 19330 raft_consensus.cc:738] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:28.695991 19885 leader_election.cc:290] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:28.674007 19330 consensus_queue.cc:260] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:28.730690 19860 leader_election.cc:290] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:28.720875 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b302fcba27244c193586bc25fc9305f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:28.752379 19192 raft_consensus.cc:3054] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:28.738898 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea78feee64340e39404cba0f1bf04db" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:28.759774 19197 raft_consensus.cc:3054] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:28.792429 19192 raft_consensus.cc:2463] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:28.809217 18630 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47522 (request call id 259) took 1025 ms. Trace:
W20250111 02:57:28.809473 18617 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet e4bd42ad66064bcaa043746081f1c759 (table default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet e4bd42ad66064bcaa043746081f1c759 consensus state has no leader
I20250111 02:57:28.812484 19197 raft_consensus.cc:2463] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:28.809587 18630 rpcz_store.cc:276] 0111 02:57:27.784291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.874303 (+ 90012us) service_pool.cc:225] Handling call
0111 02:57:27.876976 (+ 2673us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 02:57:27.878527 (+ 1551us) catalog_manager.cc:2122] Created new table and tablet info
0111 02:57:28.808525 (+929998us) catalog_manager.cc:2182] Wrote table and tablets to system table
0111 02:57:28.808876 (+ 351us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0111 02:57:28.809154 (+ 278us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:28.160813 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:28.161691 (+ 878us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:28.161716 (+ 25us) write_op.cc:435] Acquired schema lock
0111 02:57:28.161738 (+ 22us) tablet.cc:659] Decoding operations
0111 02:57:28.162145 (+ 407us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:28.162235 (+ 90us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:28.162259 (+ 24us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:28.162590 (+ 331us) tablet.cc:698] Row locks acquired
0111 02:57:28.162603 (+ 13us) write_op.cc:260] PREPARE: finished
0111 02:57:28.162943 (+ 340us) write_op.cc:270] Start()
0111 02:57:28.163108 (+ 165us) write_op.cc:276] Timestamp: P: 1736564248162865 usec, L: 0
0111 02:57:28.163126 (+ 18us) op_driver.cc:348] REPLICATION: starting
0111 02:57:28.164084 (+ 958us) log.cc:844] Serialized 433 byte log entry
0111 02:57:28.805385 (+641301us) op_driver.cc:464] REPLICATION: finished
0111 02:57:28.806602 (+ 1217us) write_op.cc:301] APPLY: starting
0111 02:57:28.806671 (+ 69us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:28.806804 (+ 133us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:28.806920 (+ 116us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:28.807431 (+ 511us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:28.807575 (+ 144us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0111 02:57:28.807624 (+ 49us) write_op.cc:312] APPLY: finished
0111 02:57:28.807832 (+ 208us) log.cc:844] Serialized 29 byte log entry
0111 02:57:28.808011 (+ 179us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:28.808092 (+ 81us) write_op.cc:454] Released schema lock
0111 02:57:28.808129 (+ 37us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":999,"apply.run_cpu_time_us":1760,"apply.run_wall_time_us":1760,"mutex_wait_us":249693,"num_ops":2,"prepare.queue_time_us":280927,"prepare.run_cpu_time_us":3861,"prepare.run_wall_time_us":130765,"replication_time_us":642089}]]}
I20250111 02:57:28.818186 18627 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet f57996ef57164187b7afbff68c1b934c
I20250111 02:57:28.818581 18627 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47522 (request call id 258) took 1165 ms. Trace:
I20250111 02:57:28.818851 18627 rpcz_store.cc:276] 0111 02:57:27.653498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.868383 (+214885us) service_pool.cc:225] Handling call
0111 02:57:27.869676 (+ 1293us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0111 02:57:27.870230 (+ 554us) catalog_manager.cc:2753] Modifying in-memory table state
0111 02:57:27.870391 (+ 161us) catalog_manager.cc:2762] Locking tablets
0111 02:57:27.870879 (+ 488us) catalog_manager.cc:2776] Removing table and tablets from system table
0111 02:57:28.817750 (+946871us) catalog_manager.cc:2794] Removing table from by-name map
0111 02:57:28.817949 (+ 199us) catalog_manager.cc:2810] Committing in-memory state
0111 02:57:28.818063 (+ 114us) catalog_manager.cc:2814] Aborting table tasks
0111 02:57:28.818524 (+ 461us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:28.292250 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:28.293458 (+ 1208us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:28.293497 (+ 39us) write_op.cc:435] Acquired schema lock
0111 02:57:28.293555 (+ 58us) tablet.cc:659] Decoding operations
0111 02:57:28.294580 (+ 1025us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:28.294617 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:28.294641 (+ 24us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:28.413955 (+119314us) tablet.cc:698] Row locks acquired
0111 02:57:28.414044 (+ 89us) write_op.cc:260] PREPARE: finished
0111 02:57:28.414344 (+ 300us) write_op.cc:270] Start()
0111 02:57:28.414703 (+ 359us) write_op.cc:276] Timestamp: P: 1736564248414204 usec, L: 0
0111 02:57:28.414776 (+ 73us) op_driver.cc:348] REPLICATION: starting
0111 02:57:28.521853 (+107077us) log.cc:844] Serialized 564 byte log entry
0111 02:57:28.815006 (+293153us) op_driver.cc:464] REPLICATION: finished
0111 02:57:28.815859 (+ 853us) write_op.cc:301] APPLY: starting
0111 02:57:28.815926 (+ 67us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:28.816031 (+ 105us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:28.816121 (+ 90us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:28.816403 (+ 282us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:28.816567 (+ 164us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0111 02:57:28.816617 (+ 50us) write_op.cc:312] APPLY: finished
0111 02:57:28.816809 (+ 192us) log.cc:844] Serialized 29 byte log entry
0111 02:57:28.816951 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:28.817026 (+ 75us) write_op.cc:454] Released schema lock
0111 02:57:28.817062 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":153,"mutex_wait_us":238731,"num_ops":2,"prepare.queue_time_us":400845,"prepare.run_cpu_time_us":6915,"prepare.run_wall_time_us":233126,"replication_time_us":400019}]]}
I20250111 02:57:28.725975 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855dac47872d4c5fbf8f5e38b588c7fe" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:28.728369 19481 leader_election.cc:290] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:28.718899 19330 ts_tablet_manager.cc:1428] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.105s user 0.014s sys 0.002s
I20250111 02:57:28.824627 19330 tablet_bootstrap.cc:492] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:28.827659 18627 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_a02982c32dcb409385f8f4e99b95423f" } new_table_name: "default.Table_6cdffa2e029846c3b95c8440c221426c" modify_external_catalogs: true
I20250111 02:57:28.863157 19321 tablet_bootstrap.cc:492] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:28.863734 19321 ts_tablet_manager.cc:1397] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.612s user 0.019s sys 0.009s
I20250111 02:57:28.866340 19321 raft_consensus.cc:357] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:28.867173 19321 raft_consensus.cc:383] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:28.867449 19321 raft_consensus.cc:738] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:28.868219 19321 consensus_queue.cc:260] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:28.870143 19321 ts_tablet_manager.cc:1428] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20250111 02:57:28.870985 19321 tablet_bootstrap.cc:492] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
W20250111 02:57:28.880185 19133 consensus_peers.cc:487] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:28.882531 19133 consensus_peers.cc:487] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fe8aa967fcb4bd7a48f6f0f69bea496. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:28.650784 19322 tablet_bootstrap.cc:492] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:28.884554 19322 ts_tablet_manager.cc:1397] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 1.589s user 0.022s sys 0.002s
I20250111 02:57:28.661357 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b302fcba27244c193586bc25fc9305f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:28.659900 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea78feee64340e39404cba0f1bf04db" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:28.886868 19322 raft_consensus.cc:357] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:28.889081 19322 raft_consensus.cc:383] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:28.889416 19322 raft_consensus.cc:738] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:28.890043 19322 consensus_queue.cc:260] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:28.892743 19322 ts_tablet_manager.cc:1428] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250111 02:57:28.893702 19322 tablet_bootstrap.cc:492] T 9d6de202f4e547a0b8b90cfa09ad319a P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:28.899291 19908 raft_consensus.cc:513] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:28.802181 18986 leader_election.cc:304] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:28.900729 18908 raft_consensus.cc:3054] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:28.901582 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 99617.1us
W20250111 02:57:28.904889 18989 leader_election.cc:343] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:28.908200 19131 leader_election.cc:343] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:28.907022 18989 leader_election.cc:343] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:28.913755 18884 consensus_meta.cc:287] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent flushing consensus metadata: real 1.135s user 0.014s sys 0.009s
I20250111 02:57:28.913924 18986 leader_election.cc:304] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:28.919759 19860 consensus_queue.cc:1035] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:28.926898 19171 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf1ce65ccf54358b95665b9764fbdb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:28.928269 19171 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b302fcba27244c193586bc25fc9305f with delete_type TABLET_DATA_DELETED (replaced by tablet 13f0a0e7698e49a5a57562dc8ad52cd5) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:28.931241 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9192160e931445f1acd207604192f759" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:28.931903 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9192160e931445f1acd207604192f759" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:28.934340 19321 tablet_bootstrap.cc:654] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:28.947813 19481 raft_consensus.cc:2798] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:28.960445 19326 ts_tablet_manager.cc:1918] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:28.961323 19849 raft_consensus.cc:2798] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:28.963915 19326 log.cc:1198] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/94b9d14959df49a6bbd80ba9e9e57ee4
I20250111 02:57:28.963861 19849 raft_consensus.cc:695] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:28.965893 19849 consensus_queue.cc:237] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:28.975445 19860 raft_consensus.cc:491] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:28.976171 19860 raft_consensus.cc:513] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:28.978780 19860 leader_election.cc:290] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:28.980755 19326 ts_tablet_manager.cc:1939] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:28.981141 19481 raft_consensus.cc:695] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
W20250111 02:57:28.985471 19133 consensus_peers.cc:487] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:28.963284 19330 tablet_bootstrap.cc:654] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:28.985214 19481 consensus_queue.cc:237] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:28.506850 18879 rpcz_store.cc:276] 0111 02:57:27.472434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.520274 (+ 47840us) service_pool.cc:225] Handling call
0111 02:57:27.549528 (+ 29254us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0111 02:57:27.549612 (+ 84us) ts_tablet_manager.cc:676] Creating new metadata...
0111 02:57:27.839042 (+289430us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:28.354674 (+515632us) tablet_replica.cc:206] Creating consensus instance
0111 02:57:28.501689 (+147015us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:28.794360 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a03880a8b0e46b7858ed1c2c2dddcaa" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:28.997092 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c6b166a29943ceb15f093fac728c79" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:28.997439 19594 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.18.20.125:40359 for RPC RequestConsensusVote timed out after 0.645s (ON_OUTBOUND_QUEUE)
I20250111 02:57:28.799226 19913 tablet.cc:1721] T 40fdec982cf643c68ef90798f4f65ad4 P 8221604733fe4fc0a1fc32de8d6af71c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250111 02:57:28.805321 18904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea78feee64340e39404cba0f1bf04db" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:28.808391 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b302fcba27244c193586bc25fc9305f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:28.820775 18905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8414653def014ce98bc38d87aac75bae" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:28.999238 18989 leader_election.cc:343] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:28.824188 18902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0158a8849bf44a4fbb5daca22883478f" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:28.999655 19131 leader_election.cc:343] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:29.000808 18695 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.18.20.126:45115 for RPC RequestConsensusVote timed out after 0.604s (ON_OUTBOUND_QUEUE)
W20250111 02:57:29.000880 18989 leader_election.cc:343] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.002583 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 167) took 4250 ms. Trace:
W20250111 02:57:28.689611 18906 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.001s sys 0.020s
I20250111 02:57:28.879084 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855dac47872d4c5fbf8f5e38b588c7fe" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:28.622684 18886 consensus_meta.cc:287] T 07f15096ea484b499fbdecec74eb6f66 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent flushing consensus metadata: real 0.801s user 0.007s sys 0.004s
I20250111 02:57:29.005407 18886 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51082 (request call id 187) took 1609 ms. Trace:
I20250111 02:57:29.005852 18886 rpcz_store.cc:276] 0111 02:57:27.396236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.514570 (+118334us) service_pool.cc:225] Handling call
0111 02:57:27.678444 (+163874us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0111 02:57:27.678518 (+ 74us) ts_tablet_manager.cc:676] Creating new metadata...
0111 02:57:27.821800 (+143282us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.004741 (+1182941us) tablet_replica.cc:206] Creating consensus instance
0111 02:57:29.005310 (+ 569us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:29.007475 18906 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.071s user 0.001s sys 0.020s
I20250111 02:57:28.715507 18882 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51082 (request call id 191) took 1060 ms. Trace:
I20250111 02:57:29.015077 18882 rpcz_store.cc:276] 0111 02:57:27.655462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.657549 (+ 2087us) service_pool.cc:225] Handling call
0111 02:57:27.658891 (+ 1342us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0111 02:57:27.658968 (+ 77us) ts_tablet_manager.cc:676] Creating new metadata...
0111 02:57:27.793415 (+134447us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:28.690534 (+897119us) tablet_replica.cc:206] Creating consensus instance
0111 02:57:28.715267 (+ 24733us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:29.003139 19326 rpcz_store.cc:276] 0111 02:57:24.751764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:24.752000 (+ 236us) service_pool.cc:225] Handling call
0111 02:57:28.179711 (+3427711us) ts_tablet_manager.cc:1145] Deleting tablet 94b9d14959df49a6bbd80ba9e9e57ee4
0111 02:57:28.179784 (+ 73us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:28.563781 (+383997us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:28.959894 (+396113us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.002449 (+ 42555us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":143612,"tablet-delete.queue_time_us":3425335}
I20250111 02:57:29.034780 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 94b9d14959df49a6bbd80ba9e9e57ee4 (table default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7]) successfully deleted
I20250111 02:57:29.035396 19326 tablet_replica.cc:331] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
W20250111 02:57:29.041203 19594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 43891.2us
I20250111 02:57:28.931675 19908 leader_election.cc:290] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:28.933574 18841 leader_election.cc:304] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:29.046486 18884 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51082 (request call id 190) took 1491 ms. Trace:
W20250111 02:57:29.047106 18841 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 113617us
I20250111 02:57:29.046891 18884 rpcz_store.cc:276] 0111 02:57:27.554796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.598862 (+ 44066us) service_pool.cc:225] Handling call
0111 02:57:27.680546 (+ 81684us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0111 02:57:27.680667 (+ 121us) ts_tablet_manager.cc:676] Creating new metadata...
0111 02:57:27.778032 (+ 97365us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.045828 (+1267796us) tablet_replica.cc:206] Creating consensus instance
0111 02:57:29.046412 (+ 584us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:29.048131 18878 consensus_meta.cc:287] T 3c57d85390324bbc9ea30a709faec01c P 8221604733fe4fc0a1fc32de8d6af71c: Time spent flushing consensus metadata: real 0.847s user 0.017s sys 0.000s
I20250111 02:57:29.049105 18878 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51082 (request call id 192) took 1393 ms. Trace:
I20250111 02:57:29.051347 19321 tablet_bootstrap.cc:492] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:29.051769 19321 ts_tablet_manager.cc:1397] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.181s user 0.009s sys 0.004s
I20250111 02:57:29.054136 19321 raft_consensus.cc:357] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.049232 19330 tablet_bootstrap.cc:492] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:29.054891 19651 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" instance_seqno: 1736564229391442) as {username='slave'} at 127.18.20.66:50733; Asking this server to re-register.
I20250111 02:57:29.055697 19330 ts_tablet_manager.cc:1397] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.231s user 0.011s sys 0.010s
I20250111 02:57:29.057528 19649 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" instance_seqno: 1736564231386866) as {username='slave'} at 127.18.20.67:41951; Asking this server to re-register.
I20250111 02:57:29.058573 19101 heartbeater.cc:463] Registering TS with master...
W20250111 02:57:29.059567 19133 leader_election.cc:343] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.060005 19133 leader_election.cc:304] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:29.060725 19860 raft_consensus.cc:2743] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:29.063581 18629 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.67:53171 (request call id 40) took 2136 ms (client timeout 1000 ms). Trace:
W20250111 02:57:29.064052 18629 rpcz_store.cc:269] 0111 02:57:26.927521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:26.993392 (+ 65871us) service_pool.cc:225] Handling call
0111 02:57:29.063501 (+2070109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6094,"reported_tablets":20}
I20250111 02:57:29.066308 18629 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_5b39045192764dd8a67ddaab57d02a5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:29.069587 19244 heartbeater.cc:463] Registering TS with master...
W20250111 02:57:29.069839 19133 leader_election.cc:343] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.070284 19133 leader_election.cc:304] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:29.070999 19860 raft_consensus.cc:2743] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:29.076210 19133 leader_election.cc:343] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.058560 19330 raft_consensus.cc:357] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.077069 19330 raft_consensus.cc:383] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:29.077067 19133 leader_election.cc:304] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:29.077394 19330 raft_consensus.cc:738] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:29.054883 19321 raft_consensus.cc:383] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:29.078061 19321 raft_consensus.cc:738] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
W20250111 02:57:29.078542 19133 leader_election.cc:343] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.079032 19133 leader_election.cc:304] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:29.079077 19330 consensus_queue.cc:260] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.079962 19860 raft_consensus.cc:2743] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:29.080574 19860 raft_consensus.cc:2743] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:29.080150 19321 consensus_queue.cc:260] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.082755 19321 ts_tablet_manager.cc:1428] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20250111 02:57:29.090354 19321 tablet_bootstrap.cc:492] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:29.091863 19682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 266 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:29.093000 19682 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c34b72b7c70343d3b6f3fae79ce4be6c in term 2.
I20250111 02:57:29.094702 18695 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 8784b9603f8544b79373158aace063af
I20250111 02:57:29.108336 19326 raft_consensus.cc:2238] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:28.899017 19322 tablet_bootstrap.cc:654] T 9d6de202f4e547a0b8b90cfa09ad319a P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:28.992974 18885 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51082 (request call id 193) took 1336 ms. Trace:
I20250111 02:57:29.058012 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c6b166a29943ceb15f093fac728c79" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:29.115581 19326 raft_consensus.cc:2267] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:29.116394 18878 rpcz_store.cc:276] 0111 02:57:27.655852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.662344 (+ 6492us) service_pool.cc:225] Handling call
0111 02:57:27.664051 (+ 1707us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0111 02:57:27.664147 (+ 96us) ts_tablet_manager.cc:676] Creating new metadata...
0111 02:57:28.186891 (+522744us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.048605 (+861714us) tablet_replica.cc:206] Creating consensus instance
0111 02:57:29.049040 (+ 435us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:29.117698 19326 ts_tablet_manager.cc:1905] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:29.075695 18622 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dea4db154779483cb069118e3f3789b0 has no consensus state
I20250111 02:57:29.126915 18622 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47522 (request call id 261) took 1309 ms. Trace:
I20250111 02:57:29.127310 18622 rpcz_store.cc:276] 0111 02:57:27.817381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.917492 (+100111us) service_pool.cc:225] Handling call
0111 02:57:27.918587 (+ 1095us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0111 02:57:27.920123 (+ 1536us) catalog_manager.cc:3627] Apply alter schema
0111 02:57:27.922268 (+ 2145us) catalog_manager.cc:3659] Acquired catalog manager lock
0111 02:57:27.922498 (+ 230us) catalog_manager.cc:3817] Updating metadata on disk
0111 02:57:29.075194 (+1152696us) catalog_manager.cc:3848] Committing alterations to in-memory state
0111 02:57:29.126832 (+ 51638us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:28.849470 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:28.849829 (+ 359us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:28.849850 (+ 21us) write_op.cc:435] Acquired schema lock
0111 02:57:28.849864 (+ 14us) tablet.cc:659] Decoding operations
0111 02:57:28.850106 (+ 242us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:28.850198 (+ 92us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:28.850221 (+ 23us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:28.850367 (+ 146us) tablet.cc:698] Row locks acquired
0111 02:57:28.850380 (+ 13us) write_op.cc:260] PREPARE: finished
0111 02:57:28.850487 (+ 107us) write_op.cc:270] Start()
0111 02:57:28.850564 (+ 77us) write_op.cc:276] Timestamp: P: 1736564248850459 usec, L: 0
0111 02:57:28.850580 (+ 16us) op_driver.cc:348] REPLICATION: starting
0111 02:57:28.851054 (+ 474us) log.cc:844] Serialized 405 byte log entry
0111 02:57:29.069279 (+218225us) op_driver.cc:464] REPLICATION: finished
0111 02:57:29.071733 (+ 2454us) write_op.cc:301] APPLY: starting
0111 02:57:29.071783 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:29.071866 (+ 83us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:29.071957 (+ 91us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:29.072115 (+ 158us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:29.072237 (+ 122us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:29.072276 (+ 39us) write_op.cc:312] APPLY: finished
0111 02:57:29.072437 (+ 161us) log.cc:844] Serialized 23 byte log entry
0111 02:57:29.072561 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:29.072615 (+ 54us) write_op.cc:454] Released schema lock
0111 02:57:29.072643 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2335,"apply.run_cpu_time_us":1094,"apply.run_wall_time_us":1094,"mutex_wait_us":220795,"num_ops":1,"prepare.queue_time_us":925764,"prepare.run_cpu_time_us":1818,"prepare.run_wall_time_us":1861,"replication_time_us":218625,"threads_started":1}]]}
I20250111 02:57:29.133723 19649 ts_manager.cc:194] Registered new tserver with Master: d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:29.135743 18629 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b39045192764dd8a67ddaab57d02a5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:29.138394 19030 tablet_service.cc:1514] Processing DeleteTablet for tablet 32afc2f158254811836fcf00f7f409ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:19 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:29.063828 19908 raft_consensus.cc:2743] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:29.089497 19330 ts_tablet_manager.cc:1428] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20250111 02:57:29.084383 18881 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51082 (request call id 186) took 1693 ms. Trace:
I20250111 02:57:29.084936 18908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.18.20.67:36199 (request call id 100) took 1069 ms. Trace:
I20250111 02:57:29.140719 19939 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250111 02:57:29.141196 19939 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:29.141702 19939 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 2 FOLLOWER]: Advancing to term 3
W20250111 02:57:29.144276 18695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.126:45115, user_credentials={real_user=slave}} blocked reactor thread for 49735.4us
I20250111 02:57:29.147812 19030 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e8c72c99ac4c369957b571d74baffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:23 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:29.149629 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 3 candidate_status { last_received { term: 2 index: 233 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:29.152197 19330 tablet_bootstrap.cc:492] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:29.158756 19330 tablet_bootstrap.cc:654] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:29.163833 19133 leader_election.cc:343] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.164403 19133 leader_election.cc:304] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:29.165184 19861 raft_consensus.cc:2743] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:29.148654 18885 rpcz_store.cc:276] 0111 02:57:27.656219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.665462 (+ 9243us) service_pool.cc:225] Handling call
0111 02:57:27.667694 (+ 2232us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0111 02:57:27.667807 (+ 113us) ts_tablet_manager.cc:676] Creating new metadata...
0111 02:57:28.199977 (+532170us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:28.992387 (+792410us) tablet_replica.cc:206] Creating consensus instance
0111 02:57:28.992852 (+ 465us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:29.172453 19321 tablet_bootstrap.cc:654] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:29.178686 19883 ts_tablet_manager.cc:1918] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:29.182170 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 3 candidate_status { last_received { term: 2 index: 233 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:29.184842 19883 log.cc:1198] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/0267a92995ea460786dc26813b5895cf
I20250111 02:57:29.185840 19883 ts_tablet_manager.cc:1939] T 0267a92995ea460786dc26813b5895cf P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
W20250111 02:57:29.186460 18621 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.66:37049 (request call id 39) took 2202 ms (client timeout 1000 ms). Trace:
I20250111 02:57:29.140108 18881 rpcz_store.cc:276] 0111 02:57:27.390951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.391088 (+ 137us) service_pool.cc:225] Handling call
0111 02:57:27.673362 (+282274us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0111 02:57:27.673477 (+ 115us) ts_tablet_manager.cc:676] Creating new metadata...
0111 02:57:28.733230 (+1059753us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.083784 (+350554us) tablet_replica.cc:206] Creating consensus instance
0111 02:57:29.084230 (+ 446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:29.197149 18906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.18.20.66:58419 (request call id 105) took 1490 ms. Trace:
I20250111 02:57:29.198069 19326 ts_tablet_manager.cc:1918] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:29.198464 19326 log.cc:1198] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/74592f80afab4513a9eb2eac351c4de5
I20250111 02:57:29.197664 18906 rpcz_store.cc:276] 0111 02:57:27.707210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.707495 (+ 285us) service_pool.cc:225] Handling call
0111 02:57:27.707969 (+ 474us) raft_consensus.cc:1466] Updating replica for 1 ops
0111 02:57:29.008680 (+1300711us) raft_consensus.cc:1529] Early marking committed up to index 0
0111 02:57:29.008727 (+ 47us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0111 02:57:29.009270 (+ 543us) log.cc:844] Serialized 26 byte log entry
0111 02:57:29.188903 (+179633us) raft_consensus.cc:1640] Marking committed up to 0
0111 02:57:29.189520 (+ 617us) raft_consensus.cc:1692] Filling consensus response to leader.
0111 02:57:29.189616 (+ 96us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0111 02:57:29.196912 (+ 7296us) raft_consensus.cc:1680] finished
0111 02:57:29.196931 (+ 19us) raft_consensus.cc:1686] UpdateReplicas() finished
0111 02:57:29.197081 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":747728,"raft.run_cpu_time_us":457,"raft.run_wall_time_us":472,"thread_start_us":1370309,"threads_started":2,"wal-append.queue_time_us":185826}
I20250111 02:57:29.199097 19326 ts_tablet_manager.cc:1939] T 74592f80afab4513a9eb2eac351c4de5 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:29.140321 18908 rpcz_store.cc:276] 0111 02:57:28.015738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:28.062536 (+ 46798us) service_pool.cc:225] Handling call
0111 02:57:28.620252 (+557716us) spinlock_profiling.cc:243] Waited 34.1 ms on lock 0x7b5800001898. stack: 00007f02faaf7853 00007f02faaf7750 00007f02fa66c584 00000000004fcd24 00000000004fccb9 00007f02fa96d2d1 00007f02faaec291 00007f02faaec142 00007f0303755cd9 00007f03029e3750 00007f0302aa97fb 00007f0302aa9881 00007f0302a6d2a0 00007f0302a587e9 00007f0302a58292 00007f0301224694
0111 02:57:28.636784 (+ 16532us) raft_consensus.cc:1466] Updating replica for 1 ops
0111 02:57:28.908862 (+272078us) raft_consensus.cc:1529] Early marking committed up to index 0
0111 02:57:28.908895 (+ 33us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0111 02:57:28.909349 (+ 454us) log.cc:844] Serialized 26 byte log entry
0111 02:57:28.953693 (+ 44344us) raft_consensus.cc:1640] Marking committed up to 1
0111 02:57:28.980205 (+ 26512us) log.cc:844] Serialized 14 byte log entry
0111 02:57:28.980700 (+ 495us) raft_consensus.cc:1692] Filling consensus response to leader.
0111 02:57:28.996414 (+ 15714us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b5400041d54. stack: 00007f02faaf7853 00007f02faaf7750 00007f02fa66c584 00000000004fcd24 00000000004fccb9 00007f03019f76d1 00007f0301a72f51 00007f0301a7199e 00007f0302a582d9 00007f0301224694 00007f0301224622 00007f0301224551 00007f03012244cc 00007f0301223782 00007f02ff83fdfc 00007f02ff83f236
0111 02:57:28.996462 (+ 48us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0111 02:57:28.996506 (+ 44us) raft_consensus.cc:1680] finished
0111 02:57:28.996528 (+ 22us) raft_consensus.cc:1686] UpdateReplicas() finished
0111 02:57:29.084866 (+ 88338us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":25240,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":181,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":251,"spinlock_wait_cycles":676397312,"thread_start_us":70104,"threads_started":2,"wal-append.queue_time_us":42353}
I20250111 02:57:29.201916 18626 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47522 (request call id 260) took 1384 ms. Trace:
I20250111 02:57:28.989599 19327 ts_tablet_manager.cc:1918] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:29.202525 19327 log.cc:1198] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/74592f80afab4513a9eb2eac351c4de5
I20250111 02:57:29.203271 19327 ts_tablet_manager.cc:1939] T 74592f80afab4513a9eb2eac351c4de5 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:29.204075 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 179) took 2079 ms. Trace:
I20250111 02:57:29.202306 18626 rpcz_store.cc:276] 0111 02:57:27.817133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.883233 (+ 66100us) service_pool.cc:225] Handling call
0111 02:57:27.892711 (+ 9478us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 02:57:27.894260 (+ 1549us) catalog_manager.cc:2122] Created new table and tablet info
0111 02:57:29.201197 (+1306937us) catalog_manager.cc:2182] Wrote table and tablets to system table
0111 02:57:29.201558 (+ 361us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0111 02:57:29.201851 (+ 293us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:28.851449 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:28.851796 (+ 347us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:28.851818 (+ 22us) write_op.cc:435] Acquired schema lock
0111 02:57:28.851830 (+ 12us) tablet.cc:659] Decoding operations
0111 02:57:28.852076 (+ 246us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:28.852110 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:28.852131 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:28.852315 (+ 184us) tablet.cc:698] Row locks acquired
0111 02:57:28.852329 (+ 14us) write_op.cc:260] PREPARE: finished
0111 02:57:28.852438 (+ 109us) write_op.cc:270] Start()
0111 02:57:28.852516 (+ 78us) write_op.cc:276] Timestamp: P: 1736564248852413 usec, L: 0
0111 02:57:28.852529 (+ 13us) op_driver.cc:348] REPLICATION: starting
0111 02:57:28.852987 (+ 458us) log.cc:844] Serialized 433 byte log entry
0111 02:57:29.155704 (+302717us) op_driver.cc:464] REPLICATION: finished
0111 02:57:29.187897 (+ 32193us) write_op.cc:301] APPLY: starting
0111 02:57:29.187956 (+ 59us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:29.191220 (+ 3264us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:29.191356 (+ 136us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:29.191874 (+ 518us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:29.192099 (+ 225us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0111 02:57:29.192161 (+ 62us) write_op.cc:312] APPLY: finished
0111 02:57:29.192419 (+ 258us) log.cc:844] Serialized 29 byte log entry
0111 02:57:29.200855 (+ 8436us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:29.200940 (+ 85us) write_op.cc:454] Released schema lock
0111 02:57:29.200972 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":31967,"mutex_wait_us":210841,"num_ops":2,"prepare.queue_time_us":917696,"prepare.run_cpu_time_us":1724,"prepare.run_wall_time_us":1722,"replication_time_us":303065}]]}
I20250111 02:57:29.181140 19651 ts_manager.cc:194] Registered new tserver with Master: 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:29.199187 18621 rpcz_store.cc:269] 0111 02:57:26.984013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.020234 (+ 36221us) service_pool.cc:225] Handling call
0111 02:57:29.186385 (+2166151us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:28.526059 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:28.847813 (+321754us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:28.847840 (+ 27us) write_op.cc:435] Acquired schema lock
0111 02:57:28.847853 (+ 13us) tablet.cc:659] Decoding operations
0111 02:57:28.848129 (+ 276us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:28.848168 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:28.848189 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:28.848344 (+ 155us) tablet.cc:698] Row locks acquired
0111 02:57:28.848358 (+ 14us) write_op.cc:260] PREPARE: finished
0111 02:57:28.848479 (+ 121us) write_op.cc:270] Start()
0111 02:57:28.848561 (+ 82us) write_op.cc:276] Timestamp: P: 1736564248848451 usec, L: 0
0111 02:57:28.848575 (+ 14us) op_driver.cc:348] REPLICATION: starting
0111 02:57:28.849112 (+ 537us) log.cc:844] Serialized 530 byte log entry
0111 02:57:29.069044 (+219932us) op_driver.cc:464] REPLICATION: finished
0111 02:57:29.070180 (+ 1136us) write_op.cc:301] APPLY: starting
0111 02:57:29.070239 (+ 59us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:29.070332 (+ 93us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:29.070435 (+ 103us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:29.070598 (+ 163us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:29.070731 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:29.070772 (+ 41us) write_op.cc:312] APPLY: finished
0111 02:57:29.070956 (+ 184us) log.cc:844] Serialized 23 byte log entry
0111 02:57:29.071161 (+ 205us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:29.071225 (+ 64us) write_op.cc:454] Released schema lock
0111 02:57:29.071256 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:29.071485 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:29.071508 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":980,"apply.run_cpu_time_us":1273,"apply.run_wall_time_us":1273,"mutex_wait_us":244638,"num_ops":1,"prepare.queue_time_us":626083,"prepare.run_cpu_time_us":3100,"prepare.run_wall_time_us":323496,"replication_time_us":220360}]]}
I20250111 02:57:29.218201 19946 tablet.cc:1721] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250111 02:57:29.184104 19594 leader_election.cc:304] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727; no voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:29.218947 19330 tablet_bootstrap.cc:492] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:29.219394 19330 ts_tablet_manager.cc:1397] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.067s user 0.012s sys 0.001s
I20250111 02:57:29.221012 19908 raft_consensus.cc:491] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:29.221498 19908 raft_consensus.cc:513] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.224812 19908 leader_election.cc:290] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:29.224679 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a983a670c2104398ac1a159ac3a1da7c" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:29.227079 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a983a670c2104398ac1a159ac3a1da7c" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:29.225751 18621 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_ccb78dfa1e94441ca883240c2d85ca5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:29.228992 18841 leader_election.cc:304] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
W20250111 02:57:29.229758 18621 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccb78dfa1e94441ca883240c2d85ca5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:29.230273 19908 raft_consensus.cc:2743] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:29.232208 19913 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.219s user 0.007s sys 0.005s
I20250111 02:57:29.227347 19321 tablet_bootstrap.cc:492] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:29.236160 19321 ts_tablet_manager.cc:1397] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.146s user 0.005s sys 0.007s
I20250111 02:57:29.204388 19327 rpcz_store.cc:276] 0111 02:57:27.124754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.124876 (+ 122us) service_pool.cc:225] Handling call
0111 02:57:27.464926 (+340050us) ts_tablet_manager.cc:1145] Deleting tablet 74592f80afab4513a9eb2eac351c4de5
0111 02:57:27.464963 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:28.451765 (+986802us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:28.989315 (+537550us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.204011 (+214696us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":339552}
I20250111 02:57:29.181670 19939 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:29.249361 19949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250111 02:57:29.251449 19594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 67460us
I20250111 02:57:29.251909 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 168) took 3986 ms. Trace:
I20250111 02:57:29.252261 19326 rpcz_store.cc:276] 0111 02:57:25.265631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:25.265847 (+ 216us) service_pool.cc:225] Handling call
0111 02:57:29.035141 (+3769294us) ts_tablet_manager.cc:1145] Deleting tablet 74592f80afab4513a9eb2eac351c4de5
0111 02:57:29.035240 (+ 99us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.168365 (+133125us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.197825 (+ 29460us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.251823 (+ 53998us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72034,"tablet-delete.queue_time_us":3768386}
I20250111 02:57:29.253623 18718 raft_consensus.cc:1235] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 3 FOLLOWER]: Rejecting Update request from peer 8784b9603f8544b79373158aace063af for earlier term 2. Current term is 3. Ops: [2.273-2.273]
I20250111 02:57:29.256244 19330 raft_consensus.cc:357] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.256979 19330 raft_consensus.cc:383] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:29.257226 19330 raft_consensus.cc:738] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:29.260372 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 169) took 2890 ms. Trace:
I20250111 02:57:29.261144 19330 consensus_queue.cc:260] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.264637 19915 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 }, Status: INVALID_TERM, Last received: 2.272, Next index: 273, Last known committed idx: 272, Time since last communication: 0.001s
I20250111 02:57:29.265733 19915 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 2 LEADER]: Stepping down as leader of term 2
I20250111 02:57:29.266134 19915 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8784b9603f8544b79373158aace063af, State: Running, Role: LEADER
I20250111 02:57:29.260721 19326 rpcz_store.cc:276] 0111 02:57:26.370330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:26.370492 (+ 162us) service_pool.cc:225] Handling call
0111 02:57:29.259914 (+2889422us) ts_tablet_manager.cc:1145] Deleting tablet 061e6eb5775443679e9cbd6b11f66fe3
0111 02:57:29.259979 (+ 65us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.260312 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2888608}
I20250111 02:57:29.271435 19327 tablet_replica.cc:331] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
W20250111 02:57:29.271407 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 061e6eb5775443679e9cbd6b11f66fe3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 061e6eb5775443679e9cbd6b11f66fe3 already in progress: creating tablet
I20250111 02:57:29.272387 19327 raft_consensus.cc:2238] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:29.272677 19330 ts_tablet_manager.cc:1428] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
I20250111 02:57:29.272871 19327 raft_consensus.cc:2267] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:29.273620 19330 tablet_bootstrap.cc:492] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:29.274624 19327 ts_tablet_manager.cc:1905] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:29.275092 19915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 2.277, Last appended by leader: 277, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:29.275781 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 170) took 2830 ms. Trace:
I20250111 02:57:29.276108 19326 rpcz_store.cc:276] 0111 02:57:26.445737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:26.445906 (+ 169us) service_pool.cc:225] Handling call
0111 02:57:29.275338 (+2829432us) ts_tablet_manager.cc:1145] Deleting tablet 42230a0bc94249fda91cf79fba48f024
0111 02:57:29.275392 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.275721 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2828629}
W20250111 02:57:29.277325 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 42230a0bc94249fda91cf79fba48f024 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42230a0bc94249fda91cf79fba48f024 already in progress: creating tablet
I20250111 02:57:29.277400 19915 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:57:29.293514 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 172) took 2713 ms. Trace:
I20250111 02:57:29.293841 19326 rpcz_store.cc:276] 0111 02:57:26.580858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:26.581000 (+ 142us) service_pool.cc:225] Handling call
0111 02:57:29.293042 (+2712042us) ts_tablet_manager.cc:1145] Deleting tablet 48dd0bccf6854d37a7ef95b5ab06308e
0111 02:57:29.293102 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.293454 (+ 352us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2710724}
I20250111 02:57:29.238690 19321 raft_consensus.cc:357] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:29.295243 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 48dd0bccf6854d37a7ef95b5ab06308e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48dd0bccf6854d37a7ef95b5ab06308e already in progress: creating tablet
I20250111 02:57:29.295542 19321 raft_consensus.cc:383] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:29.295843 19321 raft_consensus.cc:738] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:29.296594 19321 consensus_queue.cc:260] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.297435 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 272 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af"
I20250111 02:57:29.298388 18638 raft_consensus.cc:2405] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c34b72b7c70343d3b6f3fae79ce4be6c for term 3 because replica has last-logged OpId of term: 2 index: 277, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 272.
I20250111 02:57:29.298285 19682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 272 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727"
I20250111 02:57:29.298916 19682 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:57:29.299922 19939 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 3 election: Requested vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:29.188696 19883 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 202) took 1266 ms. Trace:
I20250111 02:57:29.301301 19883 rpcz_store.cc:276] 0111 02:57:27.922658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.922754 (+ 96us) service_pool.cc:225] Handling call
0111 02:57:28.007758 (+ 85004us) ts_tablet_manager.cc:1145] Deleting tablet 0267a92995ea460786dc26813b5895cf
0111 02:57:28.007862 (+ 104us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.093885 (+1086023us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.178416 (+ 84531us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.188602 (+ 10186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":268118,"tablet-delete.queue_time_us":25474,"thread_start_us":25661,"threads_started":1}
I20250111 02:57:29.303134 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 0267a92995ea460786dc26813b5895cf (table default.Table_62556c9dbbcb4e8a83e01aef82e41449 [id=e7ade2345b0845acb3ecdaa6f1ff6e7b]) successfully deleted
I20250111 02:57:29.304433 19883 tablet_replica.cc:331] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:29.305783 19883 raft_consensus.cc:2238] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:29.307119 19682 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c34b72b7c70343d3b6f3fae79ce4be6c in term 3.
I20250111 02:57:29.309666 19330 tablet_bootstrap.cc:654] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:29.309278 19947 tablet.cc:1721] T a983a670c2104398ac1a159ac3a1da7c P 54628fbcec2f4359aa9e26a734d4d29f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250111 02:57:29.311564 19326 tablet_replica.cc:331] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:29.314392 18695 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 8784b9603f8544b79373158aace063af
I20250111 02:57:29.289492 19322 tablet_bootstrap.cc:492] T 9d6de202f4e547a0b8b90cfa09ad319a P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:29.321502 19322 ts_tablet_manager.cc:1397] T 9d6de202f4e547a0b8b90cfa09ad319a P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.428s user 0.009s sys 0.008s
I20250111 02:57:29.324116 19322 raft_consensus.cc:357] T 9d6de202f4e547a0b8b90cfa09ad319a P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.324832 19322 raft_consensus.cc:383] T 9d6de202f4e547a0b8b90cfa09ad319a P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:29.325105 19322 raft_consensus.cc:738] T 9d6de202f4e547a0b8b90cfa09ad319a P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:29.325434 19939 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 3 FOLLOWER]: Leader election won for term 3
I20250111 02:57:29.329038 19322 consensus_queue.cc:260] T 9d6de202f4e547a0b8b90cfa09ad319a P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.331619 19913 tablet.cc:1721] T 9192160e931445f1acd207604192f759 P 8221604733fe4fc0a1fc32de8d6af71c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250111 02:57:29.373497 19322 ts_tablet_manager.cc:1428] T 9d6de202f4e547a0b8b90cfa09ad319a P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.052s user 0.002s sys 0.005s
I20250111 02:57:29.374586 19322 tablet_bootstrap.cc:492] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:29.375648 19951 tablet.cc:1721] T d8ef3f45e5784a29b3b32cc0de6bac84 P 8221604733fe4fc0a1fc32de8d6af71c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250111 02:57:29.370756 19651 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" instance_seqno: 1736564227335907) as {username='slave'} at 127.18.20.65:41075; Asking this server to re-register.
I20250111 02:57:29.382906 18960 heartbeater.cc:463] Registering TS with master...
I20250111 02:57:29.342247 19321 ts_tablet_manager.cc:1428] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.106s user 0.005s sys 0.000s
I20250111 02:57:29.393692 19321 tablet_bootstrap.cc:492] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:29.402395 19322 tablet_bootstrap.cc:654] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:29.318980 19326 raft_consensus.cc:2238] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:29.413347 19326 raft_consensus.cc:2267] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:29.406493 19481 raft_consensus.cc:491] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:29.421656 19481 raft_consensus.cc:513] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.381523 19883 raft_consensus.cc:2267] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:29.442489 19326 ts_tablet_manager.cc:1905] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:29.444775 19481 leader_election.cc:290] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:29.445737 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8414653def014ce98bc38d87aac75bae" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:29.446504 19197 raft_consensus.cc:2463] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:29.447902 19860 raft_consensus.cc:491] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:29.448391 19860 raft_consensus.cc:513] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.450496 19939 raft_consensus.cc:695] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 3 LEADER]: Becoming Leader. State: Replica: c34b72b7c70343d3b6f3fae79ce4be6c, State: Running, Role: LEADER
I20250111 02:57:29.451865 19860 leader_election.cc:290] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:29.451624 19939 consensus_queue.cc:237] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 2.272, Last appended by leader: 272, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:29.459688 18986 leader_election.cc:304] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:29.460657 19481 raft_consensus.cc:2798] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:29.460089 19952 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c34b72b7c70343d3b6f3fae79ce4be6c. Latest consensus state: current_term: 3 leader_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:29.460831 19327 ts_tablet_manager.cc:1918] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:29.461001 19481 raft_consensus.cc:491] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:29.461072 19952 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:29.461237 19327 log.cc:1198] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/94b9d14959df49a6bbd80ba9e9e57ee4
I20250111 02:57:29.461295 19481 raft_consensus.cc:3054] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:29.461908 19327 ts_tablet_manager.cc:1939] T 94b9d14959df49a6bbd80ba9e9e57ee4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:29.461846 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220da826a0f140d59193e8ad72096b60" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:29.462685 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 175) took 2355 ms. Trace:
W20250111 02:57:29.463670 19131 leader_election.cc:343] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.467523 19330 tablet_bootstrap.cc:492] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:29.467939 19330 ts_tablet_manager.cc:1397] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.194s user 0.007s sys 0.009s
I20250111 02:57:29.475373 19481 raft_consensus.cc:513] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.462990 19327 rpcz_store.cc:276] 0111 02:57:27.107126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.167456 (+ 60330us) service_pool.cc:225] Handling call
0111 02:57:29.271275 (+2103819us) ts_tablet_manager.cc:1145] Deleting tablet 94b9d14959df49a6bbd80ba9e9e57ee4
0111 02:57:29.271320 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.385705 (+114385us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.460582 (+ 74877us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.462630 (+ 2048us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2096559}
I20250111 02:57:29.488615 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 94b9d14959df49a6bbd80ba9e9e57ee4 (table default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7]) successfully deleted
I20250111 02:57:29.493990 19896 catalog_manager.cc:1260] Loaded cluster ID: 505f17df54394056975cd03f4ae103e8
I20250111 02:57:29.491501 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220da826a0f140d59193e8ad72096b60" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:29.491271 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8414653def014ce98bc38d87aac75bae" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:29.496269 19322 tablet_bootstrap.cc:492] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:29.496690 19322 ts_tablet_manager.cc:1397] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.122s user 0.019s sys 0.005s
W20250111 02:57:29.496892 19133 leader_election.cc:343] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.497310 19133 leader_election.cc:304] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:29.498430 19860 raft_consensus.cc:2743] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:29.479202 19330 raft_consensus.cc:357] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:29.496692 18989 leader_election.cc:343] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.489909 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 181) took 2285 ms. Trace:
I20250111 02:57:29.499639 19330 raft_consensus.cc:383] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:29.499938 19330 raft_consensus.cc:738] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:29.499518 19322 raft_consensus.cc:357] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.499675 19327 rpcz_store.cc:276] 0111 02:57:27.204557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.234653 (+ 30096us) service_pool.cc:225] Handling call
0111 02:57:29.489523 (+2254870us) ts_tablet_manager.cc:1145] Deleting tablet 2873325cece64db6a673d4fe08eb7409
0111 02:57:29.489573 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.489843 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2254187}
I20250111 02:57:29.500779 19896 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: loading cluster ID for follower catalog manager: success
I20250111 02:57:29.500671 19330 consensus_queue.cc:260] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:29.501310 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 2873325cece64db6a673d4fe08eb7409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2873325cece64db6a673d4fe08eb7409 already in progress: creating tablet
I20250111 02:57:29.503417 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 189) took 1981 ms. Trace:
I20250111 02:57:29.508548 19651 ts_manager.cc:194] Registered new tserver with Master: 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:29.511559 19849 raft_consensus.cc:491] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:29.511190 19327 rpcz_store.cc:276] 0111 02:57:27.521887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.522048 (+ 161us) service_pool.cc:225] Handling call
0111 02:57:29.503019 (+1980971us) ts_tablet_manager.cc:1145] Deleting tablet 62c6b166a29943ceb15f093fac728c79
0111 02:57:29.503095 (+ 76us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.503355 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1980260}
I20250111 02:57:29.500348 19322 raft_consensus.cc:383] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:29.512181 19322 raft_consensus.cc:738] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:29.511998 19849 raft_consensus.cc:513] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:29.512516 18617 catalog_manager.cc:4852] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): delete failed for tablet 62c6b166a29943ceb15f093fac728c79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62c6b166a29943ceb15f093fac728c79 already in progress: creating tablet
I20250111 02:57:29.512748 18869 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:51078 (request call id 53) took 1831 ms. Trace:
I20250111 02:57:29.513134 19322 consensus_queue.cc:260] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.513355 18869 rpcz_store.cc:276] 0111 02:57:27.681714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.681945 (+ 231us) service_pool.cc:225] Handling call
0111 02:57:28.827677 (+1145732us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b5400161054. stack: 00007f02faaf7853 00007f02faaf7750 00007f02fa66c584 00000000004fcd24 00000000004fccb9 00007f03019f76d1 00007f0301a6c268 00007f0302a626bd 00007f0301300e14 00007f0301300da2 00007f0301300cd1 00007f0301300c4c 00007f03012fff02 00007f02ff83fdfc 00007f02ff83f236 00007f02ff83ebdf
0111 02:57:29.226541 (+398864us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b5400191754. stack: 00007f02faaf7853 00007f02faaf7750 00007f02fa66c584 00000000004fcd24 00000000004fccb9 00007f03019f76d1 00007f0301a6c268 00007f0302a626bd 00007f0301300e14 00007f0301300da2 00007f0301300cd1 00007f0301300c4c 00007f03012fff02 00007f02ff83fdfc 00007f02ff83f236 00007f02ff83ebdf
0111 02:57:29.361046 (+134505us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b5400190d54. stack: 00007f02faaf7853 00007f02faaf7750 00007f02fa66c584 00000000004fcd24 00000000004fccb9 00007f03019f76d1 00007f0301a6c268 00007f0302a626bd 00007f0301300e14 00007f0301300da2 00007f0301300cd1 00007f0301300c4c 00007f03012fff02 00007f02ff83fdfc 00007f02ff83f236 00007f02ff83ebdf
0111 02:57:29.512567 (+151521us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1617645056}
I20250111 02:57:29.514096 19849 leader_election.cc:290] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:29.515795 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a03880a8b0e46b7858ed1c2c2dddcaa" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:29.516534 19197 raft_consensus.cc:2463] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:29.517861 18986 leader_election.cc:304] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:29.518801 19849 raft_consensus.cc:2798] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:29.519240 19849 raft_consensus.cc:491] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:29.519682 19849 raft_consensus.cc:3054] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:29.522195 19330 ts_tablet_manager.cc:1428] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.054s user 0.005s sys 0.001s
I20250111 02:57:29.522886 19923 raft_consensus.cc:491] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:29.523351 19923 raft_consensus.cc:513] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.524346 19883 ts_tablet_manager.cc:1905] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:29.525182 19923 leader_election.cc:290] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:29.526278 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf1ce65ccf54358b95665b9764fbdb0" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:29.526934 19197 raft_consensus.cc:2463] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:29.528213 18986 leader_election.cc:304] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:29.528976 19923 raft_consensus.cc:2798] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:29.529318 19923 raft_consensus.cc:491] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:29.529608 19923 raft_consensus.cc:3054] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:29.534241 19955 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:57:29.557924 19330 tablet_bootstrap.cc:492] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:29.557097 19481 leader_election.cc:290] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:29.562413 19849 raft_consensus.cc:513] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.563493 19481 raft_consensus.cc:491] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:29.566082 19481 raft_consensus.cc:513] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.577917 19321 tablet_bootstrap.cc:654] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:29.579277 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8414653def014ce98bc38d87aac75bae" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:29.579906 19197 raft_consensus.cc:3054] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:29.515934 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a03880a8b0e46b7858ed1c2c2dddcaa" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:29.523723 18903 raft_consensus.cc:3054] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:29.526690 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf1ce65ccf54358b95665b9764fbdb0" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:29.587412 18989 leader_election.cc:343] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.545320 19322 ts_tablet_manager.cc:1428] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.048s user 0.008s sys 0.000s
I20250111 02:57:29.587599 19481 leader_election.cc:290] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:29.547144 19327 tablet_replica.cc:331] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:29.576452 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8414653def014ce98bc38d87aac75bae" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:29.577184 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a03880a8b0e46b7858ed1c2c2dddcaa" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:29.588629 19327 raft_consensus.cc:2238] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:29.587584 19197 raft_consensus.cc:2463] T 8414653def014ce98bc38d87aac75bae P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:29.589159 19327 raft_consensus.cc:2267] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:29.590752 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a03880a8b0e46b7858ed1c2c2dddcaa" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:29.591347 19192 raft_consensus.cc:3054] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:29.591742 19327 ts_tablet_manager.cc:1905] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:29.593640 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b69a8c2caea4bddac3f539733bcd3af" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:29.594322 19195 raft_consensus.cc:2463] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:29.596797 19322 tablet_bootstrap.cc:492] T 1b69a8c2caea4bddac3f539733bcd3af P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:29.601914 19192 raft_consensus.cc:2463] T 6a03880a8b0e46b7858ed1c2c2dddcaa P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:29.563990 19923 raft_consensus.cc:513] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.608791 19896 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: acquiring CA information for follower catalog manager: success
I20250111 02:57:29.602663 19322 tablet_bootstrap.cc:654] T 1b69a8c2caea4bddac3f539733bcd3af P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:29.632725 19923 leader_election.cc:290] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:29.589807 18989 leader_election.cc:343] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.595640 18986 leader_election.cc:304] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
W20250111 02:57:29.634460 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 44691.8us
I20250111 02:57:29.634876 18986 leader_election.cc:304] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
W20250111 02:57:29.635099 18989 leader_election.cc:343] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.635701 18986 leader_election.cc:304] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:29.639617 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf1ce65ccf54358b95665b9764fbdb0" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:29.640179 19197 raft_consensus.cc:3054] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:29.642360 18989 leader_election.cc:343] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.564426 19849 leader_election.cc:290] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:29.643929 19923 raft_consensus.cc:2798] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:29.643828 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b69a8c2caea4bddac3f539733bcd3af" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:29.644088 19849 raft_consensus.cc:2798] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:29.644282 19923 raft_consensus.cc:491] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:29.644591 19849 raft_consensus.cc:695] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:29.643658 19481 raft_consensus.cc:2798] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:29.645298 19481 raft_consensus.cc:695] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:29.645463 19849 consensus_queue.cc:237] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.647861 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf1ce65ccf54358b95665b9764fbdb0" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:29.649286 18901 raft_consensus.cc:3054] T 9d6de202f4e547a0b8b90cfa09ad319a P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:29.652173 19330 tablet_bootstrap.cc:654] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:29.646055 19481 consensus_queue.cc:237] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.647748 19326 ts_tablet_manager.cc:1918] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:29.644610 19923 raft_consensus.cc:3054] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:29.649163 18989 leader_election.cc:343] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:29.656852 18989 leader_election.cc:343] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.656973 19321 tablet_bootstrap.cc:492] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:29.657426 19321 ts_tablet_manager.cc:1397] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.264s user 0.009s sys 0.003s
I20250111 02:57:29.657740 19326 log.cc:1198] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/199ca8692f39445f93d8c27e2134a32c
I20250111 02:57:29.662691 19326 ts_tablet_manager.cc:1939] T 199ca8692f39445f93d8c27e2134a32c P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:29.668020 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 174) took 2981 ms. Trace:
I20250111 02:57:29.668946 19197 raft_consensus.cc:2463] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:29.672091 19321 raft_consensus.cc:357] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.673029 19321 raft_consensus.cc:383] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:29.673347 19321 raft_consensus.cc:738] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:29.674119 19321 consensus_queue.cc:260] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.670123 18986 leader_election.cc:304] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:29.682818 19951 tablet.cc:1721] T a983a670c2104398ac1a159ac3a1da7c P 8221604733fe4fc0a1fc32de8d6af71c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250111 02:57:29.698206 19860 raft_consensus.cc:491] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:29.698685 19860 raft_consensus.cc:513] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:29.700501 19327 ts_tablet_manager.cc:1918] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:29.700870 19327 log.cc:1198] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/199ca8692f39445f93d8c27e2134a32c
I20250111 02:57:29.701572 19327 ts_tablet_manager.cc:1939] T 199ca8692f39445f93d8c27e2134a32c P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:29.702497 19860 leader_election.cc:290] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
W20250111 02:57:29.704803 19133 leader_election.cc:343] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: 92aa64b2de5746d1b12cc93a10c9d56e
I20250111 02:57:29.679811 19326 rpcz_store.cc:276] 0111 02:57:26.686925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:26.687369 (+ 444us) service_pool.cc:225] Handling call
0111 02:57:29.311387 (+2624018us) ts_tablet_manager.cc:1145] Deleting tablet 199ca8692f39445f93d8c27e2134a32c
0111 02:57:29.311454 (+ 67us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.478907 (+167453us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.647537 (+168630us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.667957 (+ 20420us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10973,"tablet-delete.queue_time_us":2623331}
I20250111 02:57:29.710132 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 199ca8692f39445f93d8c27e2134a32c (table default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]) successfully deleted
I20250111 02:57:29.710870 19957 raft_consensus.cc:2798] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:29.711720 19957 raft_consensus.cc:695] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
W20250111 02:57:29.712527 19131 leader_election.cc:343] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: 92aa64b2de5746d1b12cc93a10c9d56e
I20250111 02:57:29.712919 19131 leader_election.cc:304] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:29.712754 19957 consensus_queue.cc:237] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.714231 19860 raft_consensus.cc:2743] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:29.713042 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 182) took 2599 ms. Trace:
I20250111 02:57:29.715287 19326 rpcz_store.cc:276] 0111 02:57:27.113748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.133211 (+ 19463us) service_pool.cc:225] Handling call
0111 02:57:29.712671 (+2579460us) ts_tablet_manager.cc:1145] Deleting tablet 076aafe3958948079e42501c6bd86153
0111 02:57:29.712726 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.712990 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2550580}
I20250111 02:57:29.716112 19327 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51082 (request call id 194) took 2059 ms. Trace:
W20250111 02:57:29.716542 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 076aafe3958948079e42501c6bd86153 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 076aafe3958948079e42501c6bd86153 already in progress: creating tablet
I20250111 02:57:29.721590 19321 ts_tablet_manager.cc:1428] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.064s user 0.007s sys 0.000s
I20250111 02:57:29.722536 19321 tablet_bootstrap.cc:492] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:29.722105 19327 rpcz_store.cc:276] 0111 02:57:27.656585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.670575 (+ 13990us) service_pool.cc:225] Handling call
0111 02:57:29.515328 (+1844753us) ts_tablet_manager.cc:1145] Deleting tablet 199ca8692f39445f93d8c27e2134a32c
0111 02:57:29.546872 (+ 31544us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.649293 (+102421us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.700276 (+ 50983us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.716042 (+ 15766us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1843694}
I20250111 02:57:29.723790 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 199ca8692f39445f93d8c27e2134a32c (table default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]) successfully deleted
I20250111 02:57:29.677655 19923 raft_consensus.cc:513] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.730513 19860 raft_consensus.cc:491] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:29.731045 19860 raft_consensus.cc:513] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.734135 19860 leader_election.cc:290] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:29.734831 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46894710a72407993b36801f3b90eb0" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:29.736264 19133 leader_election.cc:343] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.738845 19322 tablet_bootstrap.cc:492] T 1b69a8c2caea4bddac3f539733bcd3af P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:29.741286 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46894710a72407993b36801f3b90eb0" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:29.743680 19131 leader_election.cc:343] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.744172 19131 leader_election.cc:304] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:29.746501 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 185) took 2366 ms. Trace:
I20250111 02:57:29.747231 19860 raft_consensus.cc:2743] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:29.746925 19326 rpcz_store.cc:276] 0111 02:57:27.380433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.404947 (+ 24514us) service_pool.cc:225] Handling call
0111 02:57:29.745498 (+2340551us) ts_tablet_manager.cc:1145] Deleting tablet 2873325cece64db6a673d4fe08eb7409
0111 02:57:29.745582 (+ 84us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.746275 (+ 693us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2334588}
W20250111 02:57:29.748345 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 2873325cece64db6a673d4fe08eb7409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2873325cece64db6a673d4fe08eb7409 already in progress: creating tablet
I20250111 02:57:29.749637 19896 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 02:57:29.749600 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 193) took 2319 ms. Trace:
I20250111 02:57:29.750180 19326 rpcz_store.cc:276] 0111 02:57:27.430522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.437158 (+ 6636us) service_pool.cc:225] Handling call
0111 02:57:29.749198 (+2312040us) ts_tablet_manager.cc:1145] Deleting tablet 62c6b166a29943ceb15f093fac728c79
0111 02:57:29.749259 (+ 61us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.749537 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2310605}
I20250111 02:57:29.751345 19883 ts_tablet_manager.cc:1918] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250111 02:57:29.751444 18615 catalog_manager.cc:4852] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 62c6b166a29943ceb15f093fac728c79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62c6b166a29943ceb15f093fac728c79 already in progress: creating tablet
I20250111 02:57:29.751911 19883 log.cc:1198] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/32afc2f158254811836fcf00f7f409ed
I20250111 02:57:29.752907 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 198) took 1891 ms. Trace:
I20250111 02:57:29.756345 19322 ts_tablet_manager.cc:1397] T 1b69a8c2caea4bddac3f539733bcd3af P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.160s user 0.006s sys 0.009s
I20250111 02:57:29.756317 19923 leader_election.cc:290] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:29.759634 19322 raft_consensus.cc:357] T 1b69a8c2caea4bddac3f539733bcd3af P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.759845 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b69a8c2caea4bddac3f539733bcd3af" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:29.760331 19322 raft_consensus.cc:383] T 1b69a8c2caea4bddac3f539733bcd3af P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:29.753225 19326 rpcz_store.cc:276] 0111 02:57:27.861307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.861438 (+ 131us) service_pool.cc:225] Handling call
0111 02:57:29.752606 (+1891168us) ts_tablet_manager.cc:1145] Deleting tablet 0267a92995ea460786dc26813b5895cf
0111 02:57:29.752655 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.752845 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1890554}
W20250111 02:57:29.780532 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 0267a92995ea460786dc26813b5895cf because the tablet was not found. No further retry: Not found: Tablet not found: 0267a92995ea460786dc26813b5895cf
I20250111 02:57:29.777411 19330 tablet_bootstrap.cc:492] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:29.760627 19322 raft_consensus.cc:738] T 1b69a8c2caea4bddac3f539733bcd3af P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:29.781489 19330 ts_tablet_manager.cc:1397] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.224s user 0.013s sys 0.004s
I20250111 02:57:29.785879 19322 consensus_queue.cc:260] T 1b69a8c2caea4bddac3f539733bcd3af P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.787531 18903 raft_consensus.cc:3054] T 1b69a8c2caea4bddac3f539733bcd3af P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:29.790320 19322 ts_tablet_manager.cc:1428] T 1b69a8c2caea4bddac3f539733bcd3af P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.034s user 0.003s sys 0.002s
I20250111 02:57:29.797938 18901 raft_consensus.cc:3054] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:29.799340 19322 tablet_bootstrap.cc:492] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:29.759513 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b69a8c2caea4bddac3f539733bcd3af" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:29.800220 19197 raft_consensus.cc:3054] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:29.812216 18617 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 2873325cece64db6a673d4fe08eb7409 (table default.Table_c9582d98645645f3aa80a65f7652cba8 [id=2d5d8f8a7ade443f91a7da34add25a91], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2873325cece64db6a673d4fe08eb7409 consensus state has no leader
I20250111 02:57:29.812927 19883 ts_tablet_manager.cc:1939] T 32afc2f158254811836fcf00f7f409ed P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:29.818135 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 199) took 1956 ms. Trace:
I20250111 02:57:29.818637 19326 rpcz_store.cc:276] 0111 02:57:27.862182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.862285 (+ 103us) service_pool.cc:225] Handling call
0111 02:57:29.783426 (+1921141us) ts_tablet_manager.cc:1145] Deleting tablet b54311f4269d4575b8998d7f545000e7
0111 02:57:29.817769 (+ 34343us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.818068 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1920580}
I20250111 02:57:29.820276 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 200) took 1957 ms. Trace:
W20250111 02:57:29.820488 18989 consensus_peers.cc:487] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:29.820654 19326 rpcz_store.cc:276] 0111 02:57:27.862491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.862693 (+ 202us) service_pool.cc:225] Handling call
0111 02:57:29.819964 (+1957271us) ts_tablet_manager.cc:1145] Deleting tablet d8128b3941d54ce2a72bf2ae762a774f
0111 02:57:29.820023 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.820226 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1956632}
I20250111 02:57:29.821619 19967 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:29.822100 19967 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:29.828536 19197 raft_consensus.cc:2463] T 1b69a8c2caea4bddac3f539733bcd3af P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:29.831203 19322 tablet_bootstrap.cc:654] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:29.763922 19321 tablet_bootstrap.cc:654] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:29.785014 19860 raft_consensus.cc:491] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:29.831782 19860 raft_consensus.cc:513] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.833359 19883 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 203) took 1910 ms. Trace:
I20250111 02:57:29.833608 19682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 4 candidate_status { last_received { term: 2 index: 277 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:29.834318 19682 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 3.
I20250111 02:57:29.835398 19860 leader_election.cc:290] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:29.835495 18616 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af; no voters:
I20250111 02:57:29.834854 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea76b354a75041e888cf862250e4947b" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:29.837481 19969 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250111 02:57:29.838307 19133 leader_election.cc:343] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.840453 18903 raft_consensus.cc:2463] T 1b69a8c2caea4bddac3f539733bcd3af P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:29.841076 19969 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:29.841578 19969 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:57:29.832686 19967 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:29.850473 19908 raft_consensus.cc:491] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 54628fbcec2f4359aa9e26a734d4d29f)
I20250111 02:57:29.851042 19908 raft_consensus.cc:513] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:29.854619 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet b54311f4269d4575b8998d7f545000e7 because the tablet was not found. No further retry: Not found: Tablet not found: b54311f4269d4575b8998d7f545000e7
I20250111 02:57:29.823405 19330 raft_consensus.cc:357] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.830083 18986 leader_election.cc:304] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:29.858172 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea76b354a75041e888cf862250e4947b" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:29.858256 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7158a7fc624fa4bc816ac3721c610d" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:29.859777 19131 leader_election.cc:343] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:29.859989 19908 leader_election.cc:290] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:29.860217 19131 leader_election.cc:304] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:29.861480 19860 raft_consensus.cc:2743] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:29.862660 19957 raft_consensus.cc:2798] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:29.863623 19957 raft_consensus.cc:695] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:29.865478 19322 tablet_bootstrap.cc:492] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:29.865903 19322 ts_tablet_manager.cc:1397] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.003s
I20250111 02:57:29.865509 19957 consensus_queue.cc:237] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.868245 19322 raft_consensus.cc:357] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.868914 19322 raft_consensus.cc:383] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:29.869174 19322 raft_consensus.cc:738] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:29.870766 19330 raft_consensus.cc:383] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:29.871357 19330 raft_consensus.cc:738] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:29.872556 19330 consensus_queue.cc:260] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.878643 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7158a7fc624fa4bc816ac3721c610d" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:29.880079 18841 leader_election.cc:304] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:29.881341 19908 raft_consensus.cc:2743] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:29.882187 19322 consensus_queue.cc:260] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.833741 19883 rpcz_store.cc:276] 0111 02:57:27.922930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.923044 (+ 114us) service_pool.cc:225] Handling call
0111 02:57:29.304235 (+1381191us) ts_tablet_manager.cc:1145] Deleting tablet 32afc2f158254811836fcf00f7f409ed
0111 02:57:29.304295 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.661406 (+357111us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.751085 (+ 89679us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:29.833291 (+ 82206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1320509}
I20250111 02:57:29.883985 19969 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
W20250111 02:57:29.884498 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet d8128b3941d54ce2a72bf2ae762a774f because the tablet was not found. No further retry: Not found: Tablet not found: d8128b3941d54ce2a72bf2ae762a774f
I20250111 02:57:29.886236 19682 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 3 FOLLOWER]: Refusing update from remote peer c34b72b7c70343d3b6f3fae79ce4be6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 233. Preceding OpId from leader: term: 3 index: 273. (index mismatch)
I20250111 02:57:29.886440 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 201) took 2018 ms. Trace:
I20250111 02:57:29.886833 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 32afc2f158254811836fcf00f7f409ed (table default.Table_2e1ef9ed4749430f848ae883c239e519 [id=a3db149311a04b458696fd724767db06]) successfully deleted
I20250111 02:57:29.888170 19939 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 231, Time since last communication: 0.000s
I20250111 02:57:29.889461 19322 ts_tablet_manager.cc:1428] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.023s user 0.005s sys 0.003s
I20250111 02:57:29.890383 19322 tablet_bootstrap.cc:492] T 49616e4e7cc647509e72cde57b87d6fa P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:29.897675 19682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 4 candidate_status { last_received { term: 2 index: 277 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727"
I20250111 02:57:29.898314 19682 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 3 FOLLOWER]: Advancing to term 4
W20250111 02:57:29.906921 18986 consensus_peers.cc:487] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32afc2f158254811836fcf00f7f409ed. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:29.886953 19326 rpcz_store.cc:276] 0111 02:57:27.867737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.867892 (+ 155us) service_pool.cc:225] Handling call
0111 02:57:29.884175 (+2016283us) ts_tablet_manager.cc:1145] Deleting tablet c28958a1018344a48d18e6c0f03cd0a3
0111 02:57:29.885973 (+ 1798us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.886327 (+ 354us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2015602}
W20250111 02:57:29.911334 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet c28958a1018344a48d18e6c0f03cd0a3 because the tablet was not found. No further retry: Not found: Tablet not found: c28958a1018344a48d18e6c0f03cd0a3
I20250111 02:57:29.912892 19682 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 4.
I20250111 02:57:29.914711 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 204) took 2026 ms. Trace:
I20250111 02:57:29.915462 19326 rpcz_store.cc:276] 0111 02:57:27.888061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.888483 (+ 422us) service_pool.cc:225] Handling call
0111 02:57:29.913412 (+2024929us) ts_tablet_manager.cc:1145] Deleting tablet 9fe4e60db4244a9a9696153b630c575b
0111 02:57:29.913715 (+ 303us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.914440 (+ 725us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2024092}
I20250111 02:57:29.895742 19883 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 197) took 1974 ms. Trace:
I20250111 02:57:29.917472 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 203) took 2029 ms. Trace:
I20250111 02:57:29.917279 19883 rpcz_store.cc:276] 0111 02:57:27.921110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.923146 (+ 2036us) service_pool.cc:225] Handling call
0111 02:57:29.895394 (+1972248us) ts_tablet_manager.cc:1145] Deleting tablet 74592f80afab4513a9eb2eac351c4de5
0111 02:57:29.895456 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.895670 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1911149}
I20250111 02:57:29.917831 19326 rpcz_store.cc:276] 0111 02:57:27.887787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.887916 (+ 129us) service_pool.cc:225] Handling call
0111 02:57:29.917169 (+2029253us) ts_tablet_manager.cc:1145] Deleting tablet ef70b417568049f0b496f4d8e5f7e0e4
0111 02:57:29.917225 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.917420 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2027993}
I20250111 02:57:29.921401 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 205) took 2033 ms. Trace:
I20250111 02:57:29.919138 19330 ts_tablet_manager.cc:1428] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.137s user 0.007s sys 0.000s
I20250111 02:57:29.923591 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_26836b21f0f84973b442a4966f97d1ee [id=0037843fe90a422399b837b548ff45b9]
I20250111 02:57:29.924036 19330 tablet_bootstrap.cc:492] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:29.924620 19680 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 4 FOLLOWER]: Rejecting Update request from peer c34b72b7c70343d3b6f3fae79ce4be6c for earlier term 3. Current term is 4. Ops: [2.234-3.273]
I20250111 02:57:29.926080 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609]
I20250111 02:57:29.927327 19883 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 198) took 2005 ms. Trace:
W20250111 02:57:29.918766 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 9fe4e60db4244a9a9696153b630c575b because the tablet was not found. No further retry: Not found: Tablet not found: 9fe4e60db4244a9a9696153b630c575b
I20250111 02:57:29.928264 18616 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af; no voters:
I20250111 02:57:29.913695 19969 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 4 election: Requested vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:29.929071 19967 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 4 FOLLOWER]: Leader election won for term 4
I20250111 02:57:29.927778 19883 rpcz_store.cc:276] 0111 02:57:27.921456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.923629 (+ 2173us) service_pool.cc:225] Handling call
0111 02:57:29.922927 (+1999298us) ts_tablet_manager.cc:1145] Deleting tablet 199ca8692f39445f93d8c27e2134a32c
0111 02:57:29.922989 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.927257 (+ 4268us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1938237}
I20250111 02:57:29.929520 19967 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 4 LEADER]: Becoming Leader. State: Replica: 8784b9603f8544b79373158aace063af, State: Running, Role: LEADER
W20250111 02:57:29.930235 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 199ca8692f39445f93d8c27e2134a32c because the tablet was not found. No further retry: Not found: Tablet not found: 199ca8692f39445f93d8c27e2134a32c
I20250111 02:57:29.930289 19967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 2.277, Last appended by leader: 277, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:29.935159 19321 tablet_bootstrap.cc:492] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:29.935662 19321 ts_tablet_manager.cc:1397] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.213s user 0.009s sys 0.002s
I20250111 02:57:29.941732 19883 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 199) took 2020 ms. Trace:
W20250111 02:57:29.929090 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet ef70b417568049f0b496f4d8e5f7e0e4 because the tablet was not found. No further retry: Not found: Tablet not found: ef70b417568049f0b496f4d8e5f7e0e4
I20250111 02:57:29.947381 19969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8784b9603f8544b79373158aace063af. Latest consensus state: current_term: 4 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:29.947628 19481 consensus_queue.cc:1035] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:29.954627 19326 rpcz_store.cc:276] 0111 02:57:27.889201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.889320 (+ 119us) service_pool.cc:225] Handling call
0111 02:57:29.921022 (+2031702us) ts_tablet_manager.cc:1145] Deleting tablet 1b2377893bfc426fa00d5ddacd9c5961
0111 02:57:29.921095 (+ 73us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.921341 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2030150}
I20250111 02:57:29.955628 19969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: LEADER
W20250111 02:57:29.957532 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 1b2377893bfc426fa00d5ddacd9c5961 because the tablet was not found. No further retry: Not found: Tablet not found: 1b2377893bfc426fa00d5ddacd9c5961
I20250111 02:57:29.958909 19883 rpcz_store.cc:276] 0111 02:57:27.921755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.924052 (+ 2297us) service_pool.cc:225] Handling call
0111 02:57:29.941384 (+2017332us) ts_tablet_manager.cc:1145] Deleting tablet 94b9d14959df49a6bbd80ba9e9e57ee4
0111 02:57:29.941439 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:29.941665 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1956396}
I20250111 02:57:29.956414 19939 consensus_queue.cc:1046] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 }, Status: INVALID_TERM, Last received: 2.233, Next index: 234, Last known committed idx: 231, Time since last communication: 0.000s
I20250111 02:57:29.961275 19939 raft_consensus.cc:3049] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 3 LEADER]: Stepping down as leader of term 3
I20250111 02:57:29.961676 19939 raft_consensus.cc:738] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c34b72b7c70343d3b6f3fae79ce4be6c, State: Running, Role: LEADER
I20250111 02:57:29.961972 19322 tablet_bootstrap.cc:654] T 49616e4e7cc647509e72cde57b87d6fa P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:29.962659 19939 consensus_queue.cc:260] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 3.273, Last appended by leader: 273, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:29.964327 19939 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 3 FOLLOWER]: Advancing to term 4
W20250111 02:57:29.972265 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 94b9d14959df49a6bbd80ba9e9e57ee4 because the tablet was not found. No further retry: Not found: Tablet not found: 94b9d14959df49a6bbd80ba9e9e57ee4
I20250111 02:57:29.944164 19321 raft_consensus.cc:357] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:29.973184 19321 raft_consensus.cc:383] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250111 02:57:29.973393 18989 consensus_peers.cc:487] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:29.979238 19321 raft_consensus.cc:738] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:29.966462 19326 tablet_replica.cc:331] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:29.982241 19326 raft_consensus.cc:2238] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:29.974318 19883 tablet_replica.cc:331] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:29.990149 19326 raft_consensus.cc:2267] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:29.984057 19957 raft_consensus.cc:491] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:29.990705 19957 raft_consensus.cc:513] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:29.993495 19883 raft_consensus.cc:2238] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:29.993924 19957 leader_election.cc:290] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:29.994678 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845da3ee0863430ea3fe9de8c84f2c17" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:29.996001 18989 leader_election.cc:343] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:30.005645 19326 ts_tablet_manager.cc:1905] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:29.994710 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845da3ee0863430ea3fe9de8c84f2c17" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:30.021986 19197 raft_consensus.cc:2463] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:29.979802 19330 tablet_bootstrap.cc:654] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:30.023429 18986 leader_election.cc:304] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:30.024838 19957 raft_consensus.cc:2798] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:30.025296 19957 raft_consensus.cc:491] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:30.025714 19957 raft_consensus.cc:3054] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:30.027354 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_7653ac6289d7467590ac03b43df1b8bf [id=043fd6ad4f5c4a2685af5bbecda90328]
I20250111 02:57:30.029335 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_4b9e85e01e1843ce831205177cbb2e0f [id=0a118da8c5d0408980cbd0b33e0c2ed0]
I20250111 02:57:30.030970 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_28d700d6f8444cfeab3dc52dd2bec17f [id=0e5eefce9dfa453eb277f75189def89e]
I20250111 02:57:29.995532 19321 consensus_queue.cc:260] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.050940 19330 tablet_bootstrap.cc:492] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:30.051404 19330 ts_tablet_manager.cc:1397] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.128s user 0.010s sys 0.000s
I20250111 02:57:30.053930 19330 raft_consensus.cc:357] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:30.054606 19330 raft_consensus.cc:383] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:30.054843 19330 raft_consensus.cc:738] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:30.057420 19883 raft_consensus.cc:2267] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:30.061324 19481 consensus_queue.cc:1035] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:30.080441 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_bccba04b48cc408d845624ee7ed98e5f [id=1496848ec9594cad889312978436d9c4]
I20250111 02:57:30.082171 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_c398ec562f594aca8c1e62cde38c04f9 [id=14bc02cbe30c486782d8fa4396855a9a]
I20250111 02:57:30.083815 19330 consensus_queue.cc:260] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:30.079522 19321 ts_tablet_manager.cc:1428] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.144s user 0.007s sys 0.000s
I20250111 02:57:30.097697 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_b6421db891c148c688d25610559fe16f [id=190cfd157c8d4bd19a8f40cfd80925c1]
I20250111 02:57:30.093322 19330 ts_tablet_manager.cc:1428] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.042s user 0.005s sys 0.001s
I20250111 02:57:30.101565 19330 tablet_bootstrap.cc:492] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:30.116783 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_dc22147bd8ba4ea18bc4f41b25b277c0 [id=1d14d647d8c24e31a84ce6ac7e2c217e]
I20250111 02:57:30.129283 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_37a329a4875c463ea093ad5ab23e9cf6 [id=23a0f0c1aaa149c68390a0d0327ff2c5]
I20250111 02:57:30.097002 19883 ts_tablet_manager.cc:1905] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:30.136646 19322 tablet_bootstrap.cc:492] T 49616e4e7cc647509e72cde57b87d6fa P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:30.137226 19322 ts_tablet_manager.cc:1397] T 49616e4e7cc647509e72cde57b87d6fa P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.247s user 0.012s sys 0.021s
I20250111 02:57:30.141521 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_e42719b273bf4b8f96fa48d1a3d6c8d8 [id=27f7d429d88b45b8b76d416b9dc582db]
I20250111 02:57:30.141299 19322 raft_consensus.cc:357] T 49616e4e7cc647509e72cde57b87d6fa P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.142050 19322 raft_consensus.cc:383] T 49616e4e7cc647509e72cde57b87d6fa P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:30.142328 19322 raft_consensus.cc:738] T 49616e4e7cc647509e72cde57b87d6fa P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:30.143113 19322 consensus_queue.cc:260] T 49616e4e7cc647509e72cde57b87d6fa P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.145716 19322 ts_tablet_manager.cc:1428] T 49616e4e7cc647509e72cde57b87d6fa P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250111 02:57:30.098380 19321 tablet_bootstrap.cc:492] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:30.163522 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_c9582d98645645f3aa80a65f7652cba8 [id=2d5d8f8a7ade443f91a7da34add25a91]
I20250111 02:57:30.160115 19326 ts_tablet_manager.cc:1918] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:30.174328 19326 log.cc:1198] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/5fe8aa967fcb4bd7a48f6f0f69bea496
I20250111 02:57:30.171613 19330 tablet_bootstrap.cc:654] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:30.175140 19326 ts_tablet_manager.cc:1939] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:30.184098 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 202) took 2296 ms. Trace:
I20250111 02:57:30.184459 19326 rpcz_store.cc:276] 0111 02:57:27.887491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.889333 (+ 1842us) service_pool.cc:225] Handling call
0111 02:57:29.960858 (+2071525us) ts_tablet_manager.cc:1145] Deleting tablet 5fe8aa967fcb4bd7a48f6f0f69bea496
0111 02:57:29.966163 (+ 5305us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:30.073538 (+107375us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:30.159891 (+ 86353us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:30.184036 (+ 24145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2069555}
I20250111 02:57:30.185699 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 (table default.Table_f731216a607f4fdaadd2e8dbedf7552d [id=e77c4aa40ab44fb3bc98193e503be4cf]) successfully deleted
I20250111 02:57:30.192389 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_a02982c32dcb409385f8f4e99b95423f [id=354a3a4b0b774fb9bc83623b2cb87f8a]
I20250111 02:57:30.208143 19326 tablet_replica.cc:331] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:30.208964 19326 raft_consensus.cc:2238] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:30.209901 19326 raft_consensus.cc:2267] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:30.211863 19326 ts_tablet_manager.cc:1905] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:30.208268 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]
I20250111 02:57:30.228003 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_baa2b0022e914aa5b485db90037be4af [id=3f31cb2959f340e3ae2656b63eaa52e6]
I20250111 02:57:30.231441 19322 tablet_bootstrap.cc:492] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:30.233762 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_4c9aebb06d554107a5448d968fdb0436 [id=3fab00618e6548de85b5d20adee4e9ac]
I20250111 02:57:30.235975 19321 tablet_bootstrap.cc:654] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:30.240774 19757 raft_consensus.cc:491] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:30.241343 19757 raft_consensus.cc:513] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.034617 19957 raft_consensus.cc:513] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:30.253767 19757 leader_election.cc:290] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:30.254788 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237ea33846044127987c1d81a3203553" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:30.257030 19131 leader_election.cc:343] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:30.259413 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237ea33846044127987c1d81a3203553" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:30.260102 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845da3ee0863430ea3fe9de8c84f2c17" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:30.261497 18989 leader_election.cc:343] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:30.264243 19133 leader_election.cc:343] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:30.264659 19133 leader_election.cc:304] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:30.265265 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_1ab2b1f61ced4c97ad5e44f18b6e8699 [id=40eb021e024747f0a173bf87538b7281]
I20250111 02:57:30.265797 19757 raft_consensus.cc:2743] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:30.268890 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845da3ee0863430ea3fe9de8c84f2c17" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:30.270069 19197 raft_consensus.cc:3054] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:30.277362 19197 raft_consensus.cc:2463] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:30.277871 19957 leader_election.cc:290] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:30.285429 19330 tablet_bootstrap.cc:492] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:30.286031 19330 ts_tablet_manager.cc:1397] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.185s user 0.016s sys 0.000s
I20250111 02:57:30.285940 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_bdacd508498241408a549fdbc6704512 [id=46a029216fdb4496bf9df1d776ccdf7e]
I20250111 02:57:30.289839 19330 raft_consensus.cc:357] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.298806 19330 raft_consensus.cc:383] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:30.301846 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_a68194fdb909449f878695d09b81c753 [id=49aa9df32f8e4001b66d751832a64534]
I20250111 02:57:30.304162 19322 tablet_bootstrap.cc:654] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:30.307183 19330 raft_consensus.cc:738] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:30.311228 19330 consensus_queue.cc:260] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.313818 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_bf50e4c0392544f8a24b861d3edaf46e [id=4b08fa373b0b460094c09b4d3353951c]
I20250111 02:57:30.328517 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_4f96c1244592444a994b5c5631eaaadc [id=4c817a432cf649799a2589f12a320ebd]
I20250111 02:57:30.333352 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_39f0114c91ef4d1c86ef0cf522e31e11 [id=5189b95a9b0e4b47ac0d7768e08e902b]
I20250111 02:57:30.334126 19330 ts_tablet_manager.cc:1428] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.047s user 0.004s sys 0.002s
I20250111 02:57:30.339872 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_dacea9000ff840659052c41c94a8bfaf [id=51bd1567dc9c43b2877791b72c2c2046]
I20250111 02:57:30.341665 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_535081a0b0ff46998f523371377b75f9 [id=531ca04d38e9427eb955326135436b79]
I20250111 02:57:30.347744 19322 tablet_bootstrap.cc:492] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:30.348650 19322 ts_tablet_manager.cc:1397] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.117s user 0.018s sys 0.005s
I20250111 02:57:30.360769 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_97e52a18fbd44f189d46ba9b26cd9033 [id=566e89e7552f4ce08fba16fe299b9ccf]
I20250111 02:57:30.367033 19908 raft_consensus.cc:491] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:30.367568 19908 raft_consensus.cc:513] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.369544 19908 leader_election.cc:290] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:30.374383 19321 tablet_bootstrap.cc:492] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:30.376582 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_0dcad07d56d843e59f351d933e625a21 [id=57e820dd99764a578d0c50f5f250aad4]
W20250111 02:57:30.384169 18841 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Not found: Tablet not found: 32afc2f158254811836fcf00f7f409ed
I20250111 02:57:30.382181 19321 ts_tablet_manager.cc:1397] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.284s user 0.015s sys 0.002s
I20250111 02:57:30.376238 18986 leader_election.cc:304] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:30.387311 19322 raft_consensus.cc:357] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:30.388073 19322 raft_consensus.cc:383] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:30.388247 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]
I20250111 02:57:30.388346 19322 raft_consensus.cc:738] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:30.388453 19849 raft_consensus.cc:2798] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:30.388911 19849 raft_consensus.cc:695] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:30.388983 19322 consensus_queue.cc:260] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:30.389672 19849 consensus_queue.cc:237] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:30.391160 19322 ts_tablet_manager.cc:1428] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.042s user 0.004s sys 0.004s
I20250111 02:57:30.371630 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32afc2f158254811836fcf00f7f409ed" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:30.385152 19957 raft_consensus.cc:491] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:30.394381 19957 raft_consensus.cc:513] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.396412 19957 leader_election.cc:290] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:30.399406 18842 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: Tablet not RUNNING: STOPPED
I20250111 02:57:30.399817 18842 leader_election.cc:304] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:30.402065 19908 raft_consensus.cc:2743] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:30.403052 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0158a8849bf44a4fbb5daca22883478f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:30.404824 18989 leader_election.cc:343] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:30.405272 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_85a0376bcaf24d0081e780e8c19764b8 [id=5f97e9e0988848319a9c8aaca6aad982]
I20250111 02:57:30.406239 19322 tablet_bootstrap.cc:492] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:30.406718 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_c2fa721b96ba4e9ca293ff4b5c455447 [id=64d78652fc6f44778133bd3157964db5]
I20250111 02:57:30.405598 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0158a8849bf44a4fbb5daca22883478f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:30.408051 19197 raft_consensus.cc:2463] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:30.413548 18986 leader_election.cc:304] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:30.414882 19957 raft_consensus.cc:2798] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:30.415268 19957 raft_consensus.cc:491] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:30.415566 19957 raft_consensus.cc:3054] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:30.420446 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_f36240205653470e9a67400b787ce860 [id=69f6e7b3ec40423a9d0bef5487ffe84a]
I20250111 02:57:30.361415 19330 tablet_bootstrap.cc:492] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
W20250111 02:57:30.425120 18615 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.20.124:34593 timed out after 0.580s (SENT) [suppressed 2 similar messages]
I20250111 02:57:30.435834 19322 tablet_bootstrap.cc:654] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:30.441206 19131 consensus_peers.cc:487] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa -> Peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Couldn't send request to peer 54628fbcec2f4359aa9e26a734d4d29f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fe8aa967fcb4bd7a48f6f0f69bea496. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:30.453714 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_52d947397a774f96a4fdb5087da4e19a [id=7196ac9c0c4d4dada6c94803e1220caa]
I20250111 02:57:30.457820 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_4e3a79a5d2184687a8c51fd2702c5fd0 [id=73c59c0d9864497fbc9f19e265d8283f]
I20250111 02:57:30.451692 19321 raft_consensus.cc:357] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:30.464490 19321 raft_consensus.cc:383] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:30.468847 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_b9769d513090402bb596c949defad0df [id=76e72ea89f6e408fbaeda0efc7e79770]
I20250111 02:57:30.467459 19321 raft_consensus.cc:738] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:30.476830 19923 consensus_queue.cc:1035] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:30.477979 19322 tablet_bootstrap.cc:492] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:30.478408 19322 ts_tablet_manager.cc:1397] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.072s user 0.013s sys 0.000s
I20250111 02:57:30.480808 19322 raft_consensus.cc:357] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.481480 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_6a0ed66485f640e9a2c11b62b5cbd239 [id=7b936eef4300409e908cc92b1453b500]
I20250111 02:57:30.481552 19322 raft_consensus.cc:383] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:30.481804 19322 raft_consensus.cc:738] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:30.482358 19908 raft_consensus.cc:491] T 49616e4e7cc647509e72cde57b87d6fa P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:30.482795 19908 raft_consensus.cc:513] T 49616e4e7cc647509e72cde57b87d6fa P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.484853 19908 leader_election.cc:290] T 49616e4e7cc647509e72cde57b87d6fa P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:30.486377 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49616e4e7cc647509e72cde57b87d6fa" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:30.487663 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49616e4e7cc647509e72cde57b87d6fa" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:30.490788 18842 leader_election.cc:304] T 49616e4e7cc647509e72cde57b87d6fa P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:30.476620 19321 consensus_queue.cc:260] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:30.491628 19322 consensus_queue.cc:260] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.495256 19908 raft_consensus.cc:2743] T 49616e4e7cc647509e72cde57b87d6fa P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:30.491801 19957 raft_consensus.cc:513] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.509642 19322 ts_tablet_manager.cc:1428] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.031s user 0.001s sys 0.008s
I20250111 02:57:30.511652 19883 ts_tablet_manager.cc:1918] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:57:30.514472 19322 tablet_bootstrap.cc:492] T f778763e59a94969b3f62be87e2fd867 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:30.506621 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_8eae98e1e49c4f6c9e52644256812eaa [id=7dfae414add74b6491f2f7595a551342]
I20250111 02:57:30.517314 19883 log.cc:1198] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/92aa64b2de5746d1b12cc93a10c9d56e
I20250111 02:57:30.521256 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]
I20250111 02:57:30.523241 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0158a8849bf44a4fbb5daca22883478f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:30.522322 19321 ts_tablet_manager.cc:1428] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.136s user 0.009s sys 0.007s
W20250111 02:57:30.524623 18989 leader_election.cc:343] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:30.526621 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0158a8849bf44a4fbb5daca22883478f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:30.527447 19192 raft_consensus.cc:3054] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:30.524468 19957 leader_election.cc:290] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:30.526548 19883 ts_tablet_manager.cc:1939] T 92aa64b2de5746d1b12cc93a10c9d56e P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:30.529327 19939 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:30.543341 19939 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:30.545425 19939 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:30.546239 19680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 273 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:30.546950 19680 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c34b72b7c70343d3b6f3fae79ce4be6c in term 4.
I20250111 02:57:30.548302 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_53d44be8eaeb4e59a8e3454aac56daca [id=8de61815471842e593a7d29efffc443c]
I20250111 02:57:30.548336 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 273 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:30.551224 18903 raft_consensus.cc:3054] T 49616e4e7cc647509e72cde57b87d6fa P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:30.553251 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0]
I20250111 02:57:30.549757 18695 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c; no voters:
I20250111 02:57:30.555948 19939 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250111 02:57:30.556038 19680 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 4 FOLLOWER]: Refusing update from remote peer 8784b9603f8544b79373158aace063af: Log matching property violated. Preceding OpId in replica: term: 2 index: 233. Preceding OpId from leader: term: 4 index: 278. (index mismatch)
I20250111 02:57:30.556313 19939 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:30.556602 19939 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 4 FOLLOWER]: Advancing to term 5
I20250111 02:57:30.551681 19321 tablet_bootstrap.cc:492] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:30.562086 19969 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 278, Last known committed idx: 231, Time since last communication: 0.001s
I20250111 02:57:30.563596 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_5e7c139097914dd7b7c8e55058d5290d [id=9143d1d3b8754ac78d615a1cc4601d86]
I20250111 02:57:30.566439 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605]
I20250111 02:57:30.572901 19192 raft_consensus.cc:2463] T 0158a8849bf44a4fbb5daca22883478f P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:30.573748 19939 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:30.576913 19680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 273 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727"
I20250111 02:57:30.577595 19680 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 4 FOLLOWER]: Advancing to term 5
I20250111 02:57:30.588891 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 273 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af"
I20250111 02:57:30.593998 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_c0ddb0fbf2f347c0b82acf9060722ed2 [id=964e4311f5ed4578a6f7285a9d2bbafa]
I20250111 02:57:30.596860 19326 ts_tablet_manager.cc:1918] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:30.597355 19326 log.cc:1198] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/32afc2f158254811836fcf00f7f409ed
I20250111 02:57:30.600533 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_02b5991089a34b0b8ee6039ccc092c93 [id=977f59b134154bc396bf42bc1e44e717]
I20250111 02:57:30.602115 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_98958a28faeb43e8a3b34fcf20d936aa [id=98e24956666a413cbf4564438a594ef0]
I20250111 02:57:30.595188 19939 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 5 election: Requested vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:30.605363 19322 tablet_bootstrap.cc:654] T f778763e59a94969b3f62be87e2fd867 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:30.624212 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_f9d19027fe814825a6c3906baef77ff3 [id=9a4fbc1c2c2c44c182f9935156885c5a]
I20250111 02:57:30.627233 18986 leader_election.cc:304] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:30.628891 19849 raft_consensus.cc:2798] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:30.629629 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_857f04d0de7e47a4828db9a8997f1009 [id=9cf908e0c7d44b60b88a32cf2d6368a0]
I20250111 02:57:30.640964 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7]
I20250111 02:57:30.644941 18513 master-stress-test.cc:577] Killing test thread 1/10
I20250111 02:57:30.648488 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_1431a031eea3496b9764a373a012a367 [id=a0cdf8fe647d4c84a9ce97df9ce9a0ce]
W20250111 02:57:30.655082 18989 consensus_peers.cc:487] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:30.658735 19321 tablet_bootstrap.cc:654] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:30.650873 19680 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c34b72b7c70343d3b6f3fae79ce4be6c in term 5.
I20250111 02:57:30.660131 19883 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 200) took 2737 ms. Trace:
I20250111 02:57:30.663940 18695 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 8784b9603f8544b79373158aace063af
I20250111 02:57:30.664923 19939 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 5 FOLLOWER]: Leader election won for term 5
I20250111 02:57:30.669965 19322 tablet_bootstrap.cc:492] T f778763e59a94969b3f62be87e2fd867 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:30.670430 19322 ts_tablet_manager.cc:1397] T f778763e59a94969b3f62be87e2fd867 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.159s user 0.012s sys 0.001s
I20250111 02:57:30.663836 19883 rpcz_store.cc:276] 0111 02:57:27.922086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.924743 (+ 2657us) service_pool.cc:225] Handling call
0111 02:57:29.974119 (+2049376us) ts_tablet_manager.cc:1145] Deleting tablet 92aa64b2de5746d1b12cc93a10c9d56e
0111 02:57:29.974181 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:30.297391 (+323210us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:30.505639 (+208248us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:30.659031 (+153392us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1988246}
I20250111 02:57:30.672930 19322 raft_consensus.cc:357] T f778763e59a94969b3f62be87e2fd867 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.673302 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_fc5930652905494584fde76fff8629d8 [id=a25d2996d4d64d91980bb5a067f5b118]
I20250111 02:57:30.673668 19322 raft_consensus.cc:383] T f778763e59a94969b3f62be87e2fd867 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:30.673955 19322 raft_consensus.cc:738] T f778763e59a94969b3f62be87e2fd867 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:30.666923 19682 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 5 FOLLOWER]: Rejecting Update request from peer 8784b9603f8544b79373158aace063af for earlier term 4. Current term is 5. Ops: [2.234-4.278]
I20250111 02:57:30.676265 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 92aa64b2de5746d1b12cc93a10c9d56e (table default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3]) successfully deleted
I20250111 02:57:30.679765 19969 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 }, Status: INVALID_TERM, Last received: 2.233, Next index: 234, Last known committed idx: 231, Time since last communication: 0.001s
I20250111 02:57:30.678292 19883 tablet_replica.cc:331] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:30.687729 19939 raft_consensus.cc:695] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 5 LEADER]: Becoming Leader. State: Replica: c34b72b7c70343d3b6f3fae79ce4be6c, State: Running, Role: LEADER
I20250111 02:57:30.690668 19939 consensus_queue.cc:237] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 3.273, Last appended by leader: 273, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:28.279318 19683 rpcz_store.cc:276] 0111 02:57:25.899044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:25.899346 (+ 302us) service_pool.cc:225] Handling call
0111 02:57:25.953072 (+ 53726us) raft_consensus.cc:1466] Updating replica for 228 ops
0111 02:57:26.058129 (+105057us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:26.058190 (+ 61us) write_op.cc:454] Released schema lock
0111 02:57:26.059354 (+ 1164us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:26.059413 (+ 59us) write_op.cc:454] Released schema lock
0111 02:57:26.060797 (+ 1384us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:26.060845 (+ 48us) write_op.cc:454] Released schema lock
0111 02:57:26.081431 (+ 20586us) raft_consensus.cc:1529] Early marking committed up to index 5
0111 02:57:26.103205 (+ 21774us) raft_consensus.cc:1534] Triggering prepare for 228 ops
0111 02:57:26.103807 (+ 602us) write_op.cc:270] Start()
0111 02:57:26.103940 (+ 133us) write_op.cc:276] Timestamp: P: 1736564233121915 usec, L: 0
0111 02:57:26.108086 (+ 4146us) write_op.cc:270] Start()
0111 02:57:26.108183 (+ 97us) write_op.cc:276] Timestamp: P: 1736564233131339 usec, L: 0
0111 02:57:26.128416 (+ 20233us) write_op.cc:270] Start()
0111 02:57:26.128565 (+ 149us) write_op.cc:276] Timestamp: P: 1736564233134073 usec, L: 0
0111 02:57:26.132914 (+ 4349us) write_op.cc:270] Start()
0111 02:57:26.133084 (+ 170us) write_op.cc:276] Timestamp: P: 1736564233146561 usec, L: 0
0111 02:57:26.136992 (+ 3908us) write_op.cc:270] Start()
0111 02:57:26.137179 (+ 187us) write_op.cc:276] Timestamp: P: 1736564233155309 usec, L: 0
0111 02:57:26.140550 (+ 3371us) write_op.cc:270] Start()
0111 02:57:26.140676 (+ 126us) write_op.cc:276] Timestamp: P: 1736564233160750 usec, L: 0
0111 02:57:26.141512 (+ 836us) write_op.cc:270] Start()
0111 02:57:26.141632 (+ 120us) write_op.cc:276] Timestamp: P: 1736564233183394 usec, L: 0
0111 02:57:26.142260 (+ 628us) write_op.cc:270] Start()
0111 02:57:26.142611 (+ 351us) write_op.cc:276] Timestamp: P: 1736564233186500 usec, L: 0
0111 02:57:26.143488 (+ 877us) write_op.cc:270] Start()
0111 02:57:26.143584 (+ 96us) write_op.cc:276] Timestamp: P: 1736564233238228 usec, L: 0
0111 02:57:26.144201 (+ 617us) write_op.cc:270] Start()
0111 02:57:26.144307 (+ 106us) write_op.cc:276] Timestamp: P: 1736564233275654 usec, L: 0
0111 02:57:26.144971 (+ 664us) write_op.cc:270] Start()
0111 02:57:26.145103 (+ 132us) write_op.cc:276] Timestamp: P: 1736564233341971 usec, L: 0
0111 02:57:26.145774 (+ 671us) write_op.cc:270] Start()
0111 02:57:26.145882 (+ 108us) write_op.cc:276] Timestamp: P: 1736564233367796 usec, L: 0
0111 02:57:26.146574 (+ 692us) write_op.cc:270] Start()
0111 02:57:26.146687 (+ 113us) write_op.cc:276] Timestamp: P: 1736564233427114 usec, L: 0
0111 02:57:26.148425 (+ 1738us) write_op.cc:270] Start()
0111 02:57:26.148558 (+ 133us) write_op.cc:276] Timestamp: P: 1736564233462690 usec, L: 0
0111 02:57:26.149182 (+ 624us) write_op.cc:270] Start()
0111 02:57:26.149286 (+ 104us) write_op.cc:276] Timestamp: P: 1736564233472592 usec, L: 0
0111 02:57:26.149830 (+ 544us) write_op.cc:270] Start()
0111 02:57:26.149922 (+ 92us) write_op.cc:276] Timestamp: P: 1736564233500937 usec, L: 0
0111 02:57:26.150580 (+ 658us) write_op.cc:270] Start()
0111 02:57:26.150685 (+ 105us) write_op.cc:276] Timestamp: P: 1736564233534722 usec, L: 0
0111 02:57:26.155481 (+ 4796us) write_op.cc:270] Start()
0111 02:57:26.155598 (+ 117us) write_op.cc:276] Timestamp: P: 1736564233551410 usec, L: 0
0111 02:57:26.156419 (+ 821us) write_op.cc:270] Start()
0111 02:57:26.156541 (+ 122us) write_op.cc:276] Timestamp: P: 1736564233604564 usec, L: 0
0111 02:57:26.157229 (+ 688us) write_op.cc:270] Start()
0111 02:57:26.157338 (+ 109us) write_op.cc:276] Timestamp: P: 1736564233696284 usec, L: 0
0111 02:57:26.158129 (+ 791us) write_op.cc:270] Start()
0111 02:57:26.158434 (+ 305us) write_op.cc:276] Timestamp: P: 1736564233701508 usec, L: 0
0111 02:57:26.159144 (+ 710us) write_op.cc:270] Start()
0111 02:57:26.159267 (+ 123us) write_op.cc:276] Timestamp: P: 1736564233705247 usec, L: 0
0111 02:57:26.163934 (+ 4667us) write_op.cc:270] Start()
0111 02:57:26.164108 (+ 174us) write_op.cc:276] Timestamp: P: 1736564233709047 usec, L: 0
0111 02:57:26.164917 (+ 809us) write_op.cc:270] Start()
0111 02:57:26.165039 (+ 122us) write_op.cc:276] Timestamp: P: 1736564233789913 usec, L: 0
0111 02:57:26.165702 (+ 663us) write_op.cc:270] Start()
0111 02:57:26.165817 (+ 115us) write_op.cc:276] Timestamp: P: 1736564233799780 usec, L: 0
0111 02:57:26.166506 (+ 689us) write_op.cc:270] Start()
0111 02:57:26.166627 (+ 121us) write_op.cc:276] Timestamp: P: 1736564233803996 usec, L: 0
0111 02:57:26.174642 (+ 8015us) write_op.cc:270] Start()
0111 02:57:26.174774 (+ 132us) write_op.cc:276] Timestamp: P: 1736564233815804 usec, L: 0
0111 02:57:26.175603 (+ 829us) write_op.cc:270] Start()
0111 02:57:26.175774 (+ 171us) write_op.cc:276] Timestamp: P: 1736564233834514 usec, L: 0
0111 02:57:26.176575 (+ 801us) write_op.cc:270] Start()
0111 02:57:26.176703 (+ 128us) write_op.cc:276] Timestamp: P: 1736564233935484 usec, L: 0
0111 02:57:26.179540 (+ 2837us) write_op.cc:270] Start()
0111 02:57:26.179659 (+ 119us) write_op.cc:276] Timestamp: P: 1736564234002770 usec, L: 0
0111 02:57:26.181340 (+ 1681us) write_op.cc:270] Start()
0111 02:57:26.181456 (+ 116us) write_op.cc:276] Timestamp: P: 1736564234061476 usec, L: 0
0111 02:57:26.182116 (+ 660us) write_op.cc:270] Start()
0111 02:57:26.182221 (+ 105us) write_op.cc:276] Timestamp: P: 1736564234124438 usec, L: 0
0111 02:57:26.182831 (+ 610us) write_op.cc:270] Start()
0111 02:57:26.182911 (+ 80us) write_op.cc:276] Timestamp: P: 1736564234141232 usec, L: 0
0111 02:57:26.183503 (+ 592us) write_op.cc:270] Start()
0111 02:57:26.183646 (+ 143us) write_op.cc:276] Timestamp: P: 1736564234211388 usec, L: 0
0111 02:57:26.184429 (+ 783us) write_op.cc:270] Start()
0111 02:57:26.184566 (+ 137us) write_op.cc:276] Timestamp: P: 1736564234234190 usec, L: 0
0111 02:57:26.185256 (+ 690us) write_op.cc:270] Start()
0111 02:57:26.185376 (+ 120us) write_op.cc:276] Timestamp: P: 1736564234246189 usec, L: 0
0111 02:57:26.186131 (+ 755us) write_op.cc:270] Start()
0111 02:57:26.186232 (+ 101us) write_op.cc:276] Timestamp: P: 1736564234281346 usec, L: 0
0111 02:57:26.186917 (+ 685us) write_op.cc:270] Start()
0111 02:57:26.187009 (+ 92us) write_op.cc:276] Timestamp: P: 1736564234296757 usec, L: 0
0111 02:57:26.194027 (+ 7018us) write_op.cc:270] Start()
0111 02:57:26.194147 (+ 120us) write_op.cc:276] Timestamp: P: 1736564234349575 usec, L: 0
0111 02:57:26.194870 (+ 723us) write_op.cc:270] Start()
0111 02:57:26.194978 (+ 108us) write_op.cc:276] Timestamp: P: 1736564234389678 usec, L: 0
0111 02:57:26.195840 (+ 862us) write_op.cc:270] Start()
0111 02:57:26.195988 (+ 148us) write_op.cc:276] Timestamp: P: 1736564234452943 usec, L: 0
0111 02:57:26.196855 (+ 867us) write_op.cc:270] Start()
0111 02:57:26.197006 (+ 151us) write_op.cc:276] Timestamp: P: 1736564234463008 usec, L: 0
0111 02:57:26.197826 (+ 820us) write_op.cc:270] Start()
0111 02:57:26.197964 (+ 138us) write_op.cc:276] Timestamp: P: 1736564234487388 usec, L: 0
0111 02:57:26.198785 (+ 821us) write_op.cc:270] Start()
0111 02:57:26.198930 (+ 145us) write_op.cc:276] Timestamp: P: 1736564234506665 usec, L: 0
0111 02:57:26.199763 (+ 833us) write_op.cc:270] Start()
0111 02:57:26.199907 (+ 144us) write_op.cc:276] Timestamp: P: 1736564234523307 usec, L: 0
0111 02:57:26.200721 (+ 814us) write_op.cc:270] Start()
0111 02:57:26.200852 (+ 131us) write_op.cc:276] Timestamp: P: 1736564234550667 usec, L: 0
0111 02:57:26.201702 (+ 850us) write_op.cc:270] Start()
0111 02:57:26.201983 (+ 281us) write_op.cc:276] Timestamp: P: 1736564234558408 usec, L: 0
0111 02:57:26.203103 (+ 1120us) write_op.cc:270] Start()
0111 02:57:26.203252 (+ 149us) write_op.cc:276] Timestamp: P: 1736564234622907 usec, L: 0
0111 02:57:26.203963 (+ 711us) write_op.cc:270] Start()
0111 02:57:26.204085 (+ 122us) write_op.cc:276] Timestamp: P: 1736564234721688 usec, L: 0
0111 02:57:26.204741 (+ 656us) write_op.cc:270] Start()
0111 02:57:26.204841 (+ 100us) write_op.cc:276] Timestamp: P: 1736564234803125 usec, L: 0
0111 02:57:26.205496 (+ 655us) write_op.cc:270] Start()
0111 02:57:26.205598 (+ 102us) write_op.cc:276] Timestamp: P: 1736564234860585 usec, L: 0
0111 02:57:26.206407 (+ 809us) write_op.cc:270] Start()
0111 02:57:26.206512 (+ 105us) write_op.cc:276] Timestamp: P: 1736564234864676 usec, L: 0
0111 02:57:26.207180 (+ 668us) write_op.cc:270] Start()
0111 02:57:26.207330 (+ 150us) write_op.cc:276] Timestamp: P: 1736564234889544 usec, L: 0
0111 02:57:26.208040 (+ 710us) write_op.cc:270] Start()
0111 02:57:26.208171 (+ 131us) write_op.cc:276] Timestamp: P: 1736564234948415 usec, L: 0
0111 02:57:26.208940 (+ 769us) write_op.cc:270] Start()
0111 02:57:26.209085 (+ 145us) write_op.cc:276] Timestamp: P: 1736564234959328 usec, L: 0
0111 02:57:26.210224 (+ 1139us) write_op.cc:270] Start()
0111 02:57:26.210392 (+ 168us) write_op.cc:276] Timestamp: P: 1736564235039612 usec, L: 0
0111 02:57:26.211437 (+ 1045us) write_op.cc:270] Start()
0111 02:57:26.217183 (+ 5746us) write_op.cc:276] Timestamp: P: 1736564235055545 usec, L: 0
0111 02:57:26.217985 (+ 802us) write_op.cc:270] Start()
0111 02:57:26.218109 (+ 124us) write_op.cc:276] Timestamp: P: 1736564235120158 usec, L: 0
0111 02:57:26.218822 (+ 713us) write_op.cc:270] Start()
0111 02:57:26.218930 (+ 108us) write_op.cc:276] Timestamp: P: 1736564235206302 usec, L: 0
0111 02:57:26.220099 (+ 1169us) write_op.cc:270] Start()
0111 02:57:26.223094 (+ 2995us) write_op.cc:276] Timestamp: P: 1736564235212295 usec, L: 0
0111 02:57:26.223931 (+ 837us) write_op.cc:270] Start()
0111 02:57:26.224043 (+ 112us) write_op.cc:276] Timestamp: P: 1736564235224824 usec, L: 0
0111 02:57:26.224798 (+ 755us) write_op.cc:270] Start()
0111 02:57:26.224911 (+ 113us) write_op.cc:276] Timestamp: P: 1736564235327386 usec, L: 0
0111 02:57:26.225558 (+ 647us) write_op.cc:270] Start()
0111 02:57:26.225664 (+ 106us) write_op.cc:276] Timestamp: P: 1736564235330924 usec, L: 0
0111 02:57:26.226301 (+ 637us) write_op.cc:270] Start()
0111 02:57:26.226412 (+ 111us) write_op.cc:276] Timestamp: P: 1736564235337163 usec, L: 0
0111 02:57:26.227251 (+ 839us) write_op.cc:270] Start()
0111 02:57:26.227394 (+ 143us) write_op.cc:276] Timestamp: P: 1736564235363684 usec, L: 0
0111 02:57:26.228075 (+ 681us) write_op.cc:270] Start()
0111 02:57:26.228193 (+ 118us) write_op.cc:276] Timestamp: P: 1736564235379792 usec, L: 0
0111 02:57:26.228866 (+ 673us) write_op.cc:270] Start()
0111 02:57:26.228972 (+ 106us) write_op.cc:276] Timestamp: P: 1736564235469024 usec, L: 0
0111 02:57:26.229645 (+ 673us) write_op.cc:270] Start()
0111 02:57:26.229743 (+ 98us) write_op.cc:276] Timestamp: P: 1736564235470975 usec, L: 0
0111 02:57:26.230404 (+ 661us) write_op.cc:270] Start()
0111 02:57:26.230515 (+ 111us) write_op.cc:276] Timestamp: P: 1736564235525199 usec, L: 0
0111 02:57:26.231170 (+ 655us) write_op.cc:270] Start()
0111 02:57:26.231282 (+ 112us) write_op.cc:276] Timestamp: P: 1736564235547522 usec, L: 0
0111 02:57:26.232003 (+ 721us) write_op.cc:270] Start()
0111 02:57:26.232132 (+ 129us) write_op.cc:276] Timestamp: P: 1736564235580668 usec, L: 0
0111 02:57:26.232773 (+ 641us) write_op.cc:270] Start()
0111 02:57:26.232883 (+ 110us) write_op.cc:276] Timestamp: P: 1736564235602068 usec, L: 0
0111 02:57:26.233572 (+ 689us) write_op.cc:270] Start()
0111 02:57:26.233689 (+ 117us) write_op.cc:276] Timestamp: P: 1736564235640423 usec, L: 0
0111 02:57:26.234846 (+ 1157us) write_op.cc:270] Start()
0111 02:57:26.234930 (+ 84us) write_op.cc:276] Timestamp: P: 1736564235707212 usec, L: 0
0111 02:57:26.235463 (+ 533us) write_op.cc:270] Start()
0111 02:57:26.235545 (+ 82us) write_op.cc:276] Timestamp: P: 1736564235726295 usec, L: 0
0111 02:57:26.236169 (+ 624us) write_op.cc:270] Start()
0111 02:57:26.236276 (+ 107us) write_op.cc:276] Timestamp: P: 1736564235744895 usec, L: 0
0111 02:57:26.237152 (+ 876us) write_op.cc:270] Start()
0111 02:57:26.237256 (+ 104us) write_op.cc:276] Timestamp: P: 1736564235771779 usec, L: 0
0111 02:57:26.237914 (+ 658us) write_op.cc:270] Start()
0111 02:57:26.238038 (+ 124us) write_op.cc:276] Timestamp: P: 1736564235790222 usec, L: 0
0111 02:57:26.238723 (+ 685us) write_op.cc:270] Start()
0111 02:57:26.238837 (+ 114us) write_op.cc:276] Timestamp: P: 1736564235863410 usec, L: 0
0111 02:57:26.239509 (+ 672us) write_op.cc:270] Start()
0111 02:57:26.239611 (+ 102us) write_op.cc:276] Timestamp: P: 1736564235887479 usec, L: 0
0111 02:57:26.240117 (+ 506us) write_op.cc:270] Start()
0111 02:57:26.240221 (+ 104us) write_op.cc:276] Timestamp: P: 1736564235925602 usec, L: 0
0111 02:57:26.240770 (+ 549us) write_op.cc:270] Start()
0111 02:57:26.240856 (+ 86us) write_op.cc:276] Timestamp: P: 1736564235932404 usec, L: 0
0111 02:57:26.241515 (+ 659us) write_op.cc:270] Start()
0111 02:57:26.241621 (+ 106us) write_op.cc:276] Timestamp: P: 1736564235940101 usec, L: 0
0111 02:57:26.242283 (+ 662us) write_op.cc:270] Start()
0111 02:57:26.242401 (+ 118us) write_op.cc:276] Timestamp: P: 1736564235943809 usec, L: 0
0111 02:57:26.243028 (+ 627us) write_op.cc:270] Start()
0111 02:57:26.243149 (+ 121us) write_op.cc:276] Timestamp: P: 1736564236104167 usec, L: 0
0111 02:57:26.243933 (+ 784us) write_op.cc:270] Start()
0111 02:57:26.244031 (+ 98us) write_op.cc:276] Timestamp: P: 1736564236108910 usec, L: 0
0111 02:57:26.244723 (+ 692us) write_op.cc:270] Start()
0111 02:57:26.244835 (+ 112us) write_op.cc:276] Timestamp: P: 1736564236196397 usec, L: 0
0111 02:57:26.245439 (+ 604us) write_op.cc:270] Start()
0111 02:57:26.245543 (+ 104us) write_op.cc:276] Timestamp: P: 1736564236280353 usec, L: 0
0111 02:57:26.246604 (+ 1061us) write_op.cc:270] Start()
0111 02:57:26.246714 (+ 110us) write_op.cc:276] Timestamp: P: 1736564236284692 usec, L: 0
0111 02:57:26.247506 (+ 792us) write_op.cc:270] Start()
0111 02:57:26.247606 (+ 100us) write_op.cc:276] Timestamp: P: 1736564236366991 usec, L: 0
0111 02:57:26.248259 (+ 653us) write_op.cc:270] Start()
0111 02:57:26.248370 (+ 111us) write_op.cc:276] Timestamp: P: 1736564236382803 usec, L: 0
0111 02:57:26.248980 (+ 610us) write_op.cc:270] Start()
0111 02:57:26.249263 (+ 283us) write_op.cc:276] Timestamp: P: 1736564236422477 usec, L: 0
0111 02:57:26.249990 (+ 727us) write_op.cc:270] Start()
0111 02:57:26.250113 (+ 123us) write_op.cc:276] Timestamp: P: 1736564236452011 usec, L: 0
0111 02:57:26.250831 (+ 718us) write_op.cc:270] Start()
0111 02:57:26.250947 (+ 116us) write_op.cc:276] Timestamp: P: 1736564236480431 usec, L: 0
0111 02:57:26.251658 (+ 711us) write_op.cc:270] Start()
0111 02:57:26.251778 (+ 120us) write_op.cc:276] Timestamp: P: 1736564236532330 usec, L: 0
0111 02:57:26.252474 (+ 696us) write_op.cc:270] Start()
0111 02:57:26.252572 (+ 98us) write_op.cc:276] Timestamp: P: 1736564236538252 usec, L: 0
0111 02:57:26.257575 (+ 5003us) write_op.cc:270] Start()
0111 02:57:26.257693 (+ 118us) write_op.cc:276] Timestamp: P: 1736564236545297 usec, L: 0
0111 02:57:26.259515 (+ 1822us) write_op.cc:270] Start()
0111 02:57:26.259771 (+ 256us) write_op.cc:276] Timestamp: P: 1736564236570619 usec, L: 0
0111 02:57:26.260476 (+ 705us) write_op.cc:270] Start()
0111 02:57:26.260562 (+ 86us) write_op.cc:276] Timestamp: P: 1736564236581518 usec, L: 0
0111 02:57:26.261032 (+ 470us) write_op.cc:270] Start()
0111 02:57:26.261110 (+ 78us) write_op.cc:276] Timestamp: P: 1736564236593766 usec, L: 0
0111 02:57:26.261590 (+ 480us) write_op.cc:270] Start()
0111 02:57:26.261668 (+ 78us) write_op.cc:276] Timestamp: P: 1736564236636377 usec, L: 0
0111 02:57:26.262120 (+ 452us) write_op.cc:270] Start()
0111 02:57:26.262207 (+ 87us) write_op.cc:276] Timestamp: P: 1736564236680063 usec, L: 0
0111 02:57:26.262683 (+ 476us) write_op.cc:270] Start()
0111 02:57:26.262763 (+ 80us) write_op.cc:276] Timestamp: P: 1736564236683401 usec, L: 0
0111 02:57:26.263252 (+ 489us) write_op.cc:270] Start()
0111 02:57:26.263342 (+ 90us) write_op.cc:276] Timestamp: P: 1736564236772117 usec, L: 0
0111 02:57:26.263808 (+ 466us) write_op.cc:270] Start()
0111 02:57:26.263894 (+ 86us) write_op.cc:276] Timestamp: P: 1736564236851759 usec, L: 0
0111 02:57:26.264397 (+ 503us) write_op.cc:270] Start()
0111 02:57:26.264480 (+ 83us) write_op.cc:276] Timestamp: P: 1736564236895205 usec, L: 0
0111 02:57:26.264939 (+ 459us) write_op.cc:270] Start()
0111 02:57:26.265022 (+ 83us) write_op.cc:276] Timestamp: P: 1736564236958852 usec, L: 0
0111 02:57:26.265523 (+ 501us) write_op.cc:270] Start()
0111 02:57:26.265601 (+ 78us) write_op.cc:276] Timestamp: P: 1736564237012982 usec, L: 0
0111 02:57:26.266057 (+ 456us) write_op.cc:270] Start()
0111 02:57:26.266151 (+ 94us) write_op.cc:276] Timestamp: P: 1736564237039687 usec, L: 0
0111 02:57:26.266637 (+ 486us) write_op.cc:270] Start()
0111 02:57:26.266709 (+ 72us) write_op.cc:276] Timestamp: P: 1736564237066723 usec, L: 0
0111 02:57:26.267209 (+ 500us) write_op.cc:270] Start()
0111 02:57:26.267327 (+ 118us) write_op.cc:276] Timestamp: P: 1736564237118771 usec, L: 0
0111 02:57:26.267797 (+ 470us) write_op.cc:270] Start()
0111 02:57:26.267878 (+ 81us) write_op.cc:276] Timestamp: P: 1736564237277912 usec, L: 0
0111 02:57:26.268372 (+ 494us) write_op.cc:270] Start()
0111 02:57:26.268458 (+ 86us) write_op.cc:276] Timestamp: P: 1736564237420437 usec, L: 0
0111 02:57:26.268963 (+ 505us) write_op.cc:270] Start()
0111 02:57:26.269049 (+ 86us) write_op.cc:276] Timestamp: P: 1736564237539493 usec, L: 0
0111 02:57:26.269707 (+ 658us) write_op.cc:270] Start()
0111 02:57:26.269795 (+ 88us) write_op.cc:276] Timestamp: P: 1736564237615980 usec, L: 0
0111 02:57:26.270536 (+ 741us) write_op.cc:270] Start()
0111 02:57:26.270619 (+ 83us) write_op.cc:276] Timestamp: P: 1736564237656957 usec, L: 0
0111 02:57:26.271161 (+ 542us) write_op.cc:270] Start()
0111 02:57:26.271251 (+ 90us) write_op.cc:276] Timestamp: P: 1736564237724230 usec, L: 0
0111 02:57:26.271750 (+ 499us) write_op.cc:270] Start()
0111 02:57:26.271837 (+ 87us) write_op.cc:276] Timestamp: P: 1736564237830458 usec, L: 0
0111 02:57:26.272339 (+ 502us) write_op.cc:270] Start()
0111 02:57:26.272425 (+ 86us) write_op.cc:276] Timestamp: P: 1736564237959762 usec, L: 0
0111 02:57:26.272905 (+ 480us) write_op.cc:270] Start()
0111 02:57:26.272983 (+ 78us) write_op.cc:276] Timestamp: P: 1736564238161034 usec, L: 0
0111 02:57:26.273471 (+ 488us) write_op.cc:270] Start()
0111 02:57:26.273549 (+ 78us) write_op.cc:276] Timestamp: P: 1736564238162819 usec, L: 0
0111 02:57:26.273999 (+ 450us) write_op.cc:270] Start()
0111 02:57:26.274069 (+ 70us) write_op.cc:276] Timestamp: P: 1736564238165147 usec, L: 0
0111 02:57:26.274600 (+ 531us) write_op.cc:270] Start()
0111 02:57:26.274687 (+ 87us) write_op.cc:276] Timestamp: P: 1736564238168169 usec, L: 0
0111 02:57:26.275240 (+ 553us) write_op.cc:270] Start()
0111 02:57:26.275359 (+ 119us) write_op.cc:276] Timestamp: P: 1736564238170256 usec, L: 0
0111 02:57:26.275920 (+ 561us) write_op.cc:270] Start()
0111 02:57:26.275997 (+ 77us) write_op.cc:276] Timestamp: P: 1736564238244106 usec, L: 0
0111 02:57:26.276459 (+ 462us) write_op.cc:270] Start()
0111 02:57:26.276531 (+ 72us) write_op.cc:276] Timestamp: P: 1736564238332109 usec, L: 0
0111 02:57:26.276982 (+ 451us) write_op.cc:270] Start()
0111 02:57:26.277056 (+ 74us) write_op.cc:276] Timestamp: P: 1736564238357844 usec, L: 0
0111 02:57:26.277588 (+ 532us) write_op.cc:270] Start()
0111 02:57:26.277664 (+ 76us) write_op.cc:276] Timestamp: P: 1736564238456261 usec, L: 0
0111 02:57:26.278415 (+ 751us) write_op.cc:270] Start()
0111 02:57:26.278496 (+ 81us) write_op.cc:276] Timestamp: P: 1736564238474346 usec, L: 0
0111 02:57:26.278956 (+ 460us) write_op.cc:270] Start()
0111 02:57:26.279041 (+ 85us) write_op.cc:276] Timestamp: P: 1736564238478346 usec, L: 0
0111 02:57:26.279602 (+ 561us) write_op.cc:270] Start()
0111 02:57:26.279677 (+ 75us) write_op.cc:276] Timestamp: P: 1736564238498379 usec, L: 0
0111 02:57:26.280149 (+ 472us) write_op.cc:270] Start()
0111 02:57:26.280227 (+ 78us) write_op.cc:276] Timestamp: P: 1736564238628675 usec, L: 0
0111 02:57:26.280725 (+ 498us) write_op.cc:270] Start()
0111 02:57:26.280803 (+ 78us) write_op.cc:276] Timestamp: P: 1736564238660128 usec, L: 0
0111 02:57:26.281419 (+ 616us) write_op.cc:270] Start()
0111 02:57:26.281508 (+ 89us) write_op.cc:276] Timestamp: P: 1736564238762291 usec, L: 0
0111 02:57:26.281992 (+ 484us) write_op.cc:270] Start()
0111 02:57:26.282073 (+ 81us) write_op.cc:276] Timestamp: P: 1736564238840156 usec, L: 0
0111 02:57:26.282599 (+ 526us) write_op.cc:270] Start()
0111 02:57:26.282689 (+ 90us) write_op.cc:276] Timestamp: P: 1736564238846211 usec, L: 0
0111 02:57:26.283226 (+ 537us) write_op.cc:270] Start()
0111 02:57:26.283329 (+ 103us) write_op.cc:276] Timestamp: P: 1736564238883063 usec, L: 0
0111 02:57:26.283853 (+ 524us) write_op.cc:270] Start()
0111 02:57:26.283954 (+ 101us) write_op.cc:276] Timestamp: P: 1736564238887581 usec, L: 0
0111 02:57:26.284454 (+ 500us) write_op.cc:270] Start()
0111 02:57:26.284536 (+ 82us) write_op.cc:276] Timestamp: P: 1736564239018450 usec, L: 0
0111 02:57:26.284994 (+ 458us) write_op.cc:270] Start()
0111 02:57:26.285070 (+ 76us) write_op.cc:276] Timestamp: P: 1736564239229411 usec, L: 0
0111 02:57:26.285546 (+ 476us) write_op.cc:270] Start()
0111 02:57:26.285646 (+ 100us) write_op.cc:276] Timestamp: P: 1736564239286544 usec, L: 0
0111 02:57:26.286126 (+ 480us) write_op.cc:270] Start()
0111 02:57:26.286202 (+ 76us) write_op.cc:276] Timestamp: P: 1736564239561760 usec, L: 0
0111 02:57:26.286609 (+ 407us) write_op.cc:270] Start()
0111 02:57:26.286676 (+ 67us) write_op.cc:276] Timestamp: P: 1736564239566451 usec, L: 0
0111 02:57:26.287147 (+ 471us) write_op.cc:270] Start()
0111 02:57:26.287216 (+ 69us) write_op.cc:276] Timestamp: P: 1736564239651896 usec, L: 0
0111 02:57:26.287688 (+ 472us) write_op.cc:270] Start()
0111 02:57:26.287761 (+ 73us) write_op.cc:276] Timestamp: P: 1736564239696026 usec, L: 0
0111 02:57:26.288172 (+ 411us) write_op.cc:270] Start()
0111 02:57:26.288240 (+ 68us) write_op.cc:276] Timestamp: P: 1736564239725849 usec, L: 0
0111 02:57:26.288854 (+ 614us) write_op.cc:270] Start()
0111 02:57:26.288920 (+ 66us) write_op.cc:276] Timestamp: P: 1736564239727958 usec, L: 0
0111 02:57:26.289320 (+ 400us) write_op.cc:270] Start()
0111 02:57:26.289385 (+ 65us) write_op.cc:276] Timestamp: P: 1736564239730046 usec, L: 0
0111 02:57:26.289774 (+ 389us) write_op.cc:270] Start()
0111 02:57:26.289836 (+ 62us) write_op.cc:276] Timestamp: P: 1736564239946909 usec, L: 0
0111 02:57:26.290376 (+ 540us) write_op.cc:270] Start()
0111 02:57:26.290440 (+ 64us) write_op.cc:276] Timestamp: P: 1736564239956184 usec, L: 0
0111 02:57:26.290855 (+ 415us) write_op.cc:270] Start()
0111 02:57:26.290928 (+ 73us) write_op.cc:276] Timestamp: P: 1736564239968294 usec, L: 0
0111 02:57:26.291399 (+ 471us) write_op.cc:270] Start()
0111 02:57:26.291466 (+ 67us) write_op.cc:276] Timestamp: P: 1736564239978613 usec, L: 0
0111 02:57:26.291841 (+ 375us) write_op.cc:270] Start()
0111 02:57:26.291905 (+ 64us) write_op.cc:276] Timestamp: P: 1736564240032232 usec, L: 0
0111 02:57:26.292308 (+ 403us) write_op.cc:270] Start()
0111 02:57:26.292372 (+ 64us) write_op.cc:276] Timestamp: P: 1736564240071396 usec, L: 0
0111 02:57:26.292778 (+ 406us) write_op.cc:270] Start()
0111 02:57:26.292845 (+ 67us) write_op.cc:276] Timestamp: P: 1736564240091082 usec, L: 0
0111 02:57:26.293233 (+ 388us) write_op.cc:270] Start()
0111 02:57:26.293309 (+ 76us) write_op.cc:276] Timestamp: P: 1736564240151889 usec, L: 0
0111 02:57:26.293734 (+ 425us) write_op.cc:270] Start()
0111 02:57:26.293803 (+ 69us) write_op.cc:276] Timestamp: P: 1736564240215984 usec, L: 0
0111 02:57:26.294198 (+ 395us) write_op.cc:270] Start()
0111 02:57:26.294289 (+ 91us) write_op.cc:276] Timestamp: P: 1736564240244294 usec, L: 0
0111 02:57:26.294689 (+ 400us) write_op.cc:270] Start()
0111 02:57:26.294768 (+ 79us) write_op.cc:276] Timestamp: P: 1736564240268551 usec, L: 0
0111 02:57:26.295214 (+ 446us) write_op.cc:270] Start()
0111 02:57:26.295292 (+ 78us) write_op.cc:276] Timestamp: P: 1736564240294408 usec, L: 0
0111 02:57:26.295696 (+ 404us) write_op.cc:270] Start()
0111 02:57:26.295789 (+ 93us) write_op.cc:276] Timestamp: P: 1736564240300529 usec, L: 0
0111 02:57:26.339610 (+ 43821us) write_op.cc:270] Start()
0111 02:57:26.339723 (+ 113us) write_op.cc:276] Timestamp: P: 1736564240348004 usec, L: 0
0111 02:57:26.340323 (+ 600us) write_op.cc:270] Start()
0111 02:57:26.340513 (+ 190us) write_op.cc:276] Timestamp: P: 1736564240396804 usec, L: 0
0111 02:57:26.340989 (+ 476us) write_op.cc:270] Start()
0111 02:57:26.341061 (+ 72us) write_op.cc:276] Timestamp: P: 1736564240439039 usec, L: 0
0111 02:57:26.341545 (+ 484us) write_op.cc:270] Start()
0111 02:57:26.341612 (+ 67us) write_op.cc:276] Timestamp: P: 1736564240509221 usec, L: 0
0111 02:57:26.342036 (+ 424us) write_op.cc:270] Start()
0111 02:57:26.342108 (+ 72us) write_op.cc:276] Timestamp: P: 1736564240557718 usec, L: 0
0111 02:57:26.342540 (+ 432us) write_op.cc:270] Start()
0111 02:57:26.342662 (+ 122us) write_op.cc:276] Timestamp: P: 1736564240561501 usec, L: 0
0111 02:57:26.343265 (+ 603us) write_op.cc:270] Start()
0111 02:57:26.343371 (+ 106us) write_op.cc:276] Timestamp: P: 1736564240635431 usec, L: 0
0111 02:57:26.344124 (+ 753us) write_op.cc:270] Start()
0111 02:57:26.344225 (+ 101us) write_op.cc:276] Timestamp: P: 1736564240642963 usec, L: 0
0111 02:57:26.344802 (+ 577us) write_op.cc:270] Start()
0111 02:57:26.344868 (+ 66us) write_op.cc:276] Timestamp: P: 1736564240749371 usec, L: 0
0111 02:57:26.345296 (+ 428us) write_op.cc:270] Start()
0111 02:57:26.345378 (+ 82us) write_op.cc:276] Timestamp: P: 1736564240826826 usec, L: 0
0111 02:57:26.345823 (+ 445us) write_op.cc:270] Start()
0111 02:57:26.345894 (+ 71us) write_op.cc:276] Timestamp: P: 1736564240853145 usec, L: 0
0111 02:57:26.346339 (+ 445us) write_op.cc:270] Start()
0111 02:57:26.346415 (+ 76us) write_op.cc:276] Timestamp: P: 1736564241004038 usec, L: 0
0111 02:57:26.346859 (+ 444us) write_op.cc:270] Start()
0111 02:57:26.346932 (+ 73us) write_op.cc:276] Timestamp: P: 1736564241005966 usec, L: 0
0111 02:57:26.443295 (+ 96363us) write_op.cc:270] Start()
0111 02:57:26.443418 (+ 123us) write_op.cc:276] Timestamp: P: 1736564241032073 usec, L: 0
0111 02:57:26.444056 (+ 638us) write_op.cc:270] Start()
0111 02:57:26.444131 (+ 75us) write_op.cc:276] Timestamp: P: 1736564241070608 usec, L: 0
0111 02:57:26.444636 (+ 505us) write_op.cc:270] Start()
0111 02:57:26.444708 (+ 72us) write_op.cc:276] Timestamp: P: 1736564241122028 usec, L: 0
0111 02:57:26.445191 (+ 483us) write_op.cc:270] Start()
0111 02:57:26.445264 (+ 73us) write_op.cc:276] Timestamp: P: 1736564241231683 usec, L: 0
0111 02:57:26.445714 (+ 450us) write_op.cc:270] Start()
0111 02:57:26.445785 (+ 71us) write_op.cc:276] Timestamp: P: 1736564241300618 usec, L: 0
0111 02:57:26.446217 (+ 432us) write_op.cc:270] Start()
0111 02:57:26.446289 (+ 72us) write_op.cc:276] Timestamp: P: 1736564241302533 usec, L: 0
0111 02:57:26.446723 (+ 434us) write_op.cc:270] Start()
0111 02:57:26.446789 (+ 66us) write_op.cc:276] Timestamp: P: 1736564241315187 usec, L: 0
0111 02:57:26.453427 (+ 6638us) write_op.cc:270] Start()
0111 02:57:26.453514 (+ 87us) write_op.cc:276] Timestamp: P: 1736564241335551 usec, L: 0
0111 02:57:26.454030 (+ 516us) write_op.cc:270] Start()
0111 02:57:26.454171 (+ 141us) write_op.cc:276] Timestamp: P: 1736564241373194 usec, L: 0
0111 02:57:26.454678 (+ 507us) write_op.cc:270] Start()
0111 02:57:26.454748 (+ 70us) write_op.cc:276] Timestamp: P: 1736564241428692 usec, L: 0
0111 02:57:26.455350 (+ 602us) write_op.cc:270] Start()
0111 02:57:26.455421 (+ 71us) write_op.cc:276] Timestamp: P: 1736564241445384 usec, L: 0
0111 02:57:26.455854 (+ 433us) write_op.cc:270] Start()
0111 02:57:26.455927 (+ 73us) write_op.cc:276] Timestamp: P: 1736564241477087 usec, L: 0
0111 02:57:26.457752 (+ 1825us) write_op.cc:270] Start()
0111 02:57:26.457838 (+ 86us) write_op.cc:276] Timestamp: P: 1736564241645688 usec, L: 0
0111 02:57:26.458366 (+ 528us) write_op.cc:270] Start()
0111 02:57:26.458451 (+ 85us) write_op.cc:276] Timestamp: P: 1736564242012603 usec, L: 0
0111 02:57:26.459048 (+ 597us) write_op.cc:270] Start()
0111 02:57:26.459153 (+ 105us) write_op.cc:276] Timestamp: P: 1736564242059313 usec, L: 0
0111 02:57:26.565026 (+105873us) write_op.cc:270] Start()
0111 02:57:26.565126 (+ 100us) write_op.cc:276] Timestamp: P: 1736564242211937 usec, L: 0
0111 02:57:26.566460 (+ 1334us) write_op.cc:270] Start()
0111 02:57:26.566552 (+ 92us) write_op.cc:276] Timestamp: P: 1736564242220157 usec, L: 0
0111 02:57:26.571765 (+ 5213us) write_op.cc:270] Start()
0111 02:57:26.571845 (+ 80us) write_op.cc:276] Timestamp: P: 1736564242306467 usec, L: 0
0111 02:57:26.572298 (+ 453us) write_op.cc:270] Start()
0111 02:57:26.572364 (+ 66us) write_op.cc:276] Timestamp: P: 1736564242315962 usec, L: 0
0111 02:57:26.572779 (+ 415us) write_op.cc:270] Start()
0111 02:57:26.572845 (+ 66us)
I20250111 02:57:30.699888 19322 consensus_queue.cc:260] T f778763e59a94969b3f62be87e2fd867 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.689746 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_bdf982e1d6904892ac08ff16ed20a636 [id=a2b6467b58754b85ab0481a9e8aa0ace]
I20250111 02:57:30.702394 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19]
I20250111 02:57:30.705085 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_ee18021e6a544ed6b86b624aec8c39be [id=a9538f81833c4c0d8d02b6271a67f711]
W20250111 02:57:30.705343 18989 consensus_peers.cc:487] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:30.710637 19322 ts_tablet_manager.cc:1428] T f778763e59a94969b3f62be87e2fd867 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.040s user 0.007s sys 0.002s
I20250111 02:57:30.711673 19322 tablet_bootstrap.cc:492] T 845da3ee0863430ea3fe9de8c84f2c17 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:30.715373 19995 consensus_queue.cc:1035] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:30.717516 19322 tablet_bootstrap.cc:654] T 845da3ee0863430ea3fe9de8c84f2c17 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:30.727990 19326 ts_tablet_manager.cc:1939] T 32afc2f158254811836fcf00f7f409ed P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:30.621851 19330 tablet_bootstrap.cc:654] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:30.736831 19849 raft_consensus.cc:695] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:30.738972 19849 consensus_queue.cc:237] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.731029 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_2c181f66f28741b7b388a7b563b7d2f5 [id=ac0e88a97cf740fd8df3e6cbd81888b5]
I20250111 02:57:30.758422 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3]
I20250111 02:57:30.761621 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_6ad0b827920c42c68113310190019df1 [id=b25314ef99fc4e0ba80c6391223d2a3d]
I20250111 02:57:30.762050 18901 raft_consensus.cc:3054] T f778763e59a94969b3f62be87e2fd867 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:30.767004 18989 consensus_peers.cc:487] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:30.771800 19957 raft_consensus.cc:491] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:30.772315 19957 raft_consensus.cc:513] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:30.774901 19957 leader_election.cc:290] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:30.776376 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855dac47872d4c5fbf8f5e38b588c7fe" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:30.778220 18989 leader_election.cc:343] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:30.785766 19998 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 4 LEADER]: Stepping down as leader of term 4
I20250111 02:57:30.786206 19998 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8784b9603f8544b79373158aace063af, State: Running, Role: LEADER
I20250111 02:57:30.739495 19883 raft_consensus.cc:2238] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:30.782415 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855dac47872d4c5fbf8f5e38b588c7fe" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:30.788951 19883 raft_consensus.cc:2267] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:30.788945 19195 raft_consensus.cc:2463] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:30.790494 18986 leader_election.cc:304] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:30.791349 19957 raft_consensus.cc:2798] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:30.791685 19957 raft_consensus.cc:491] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:30.792003 19957 raft_consensus.cc:3054] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:30.797000 19322 tablet_bootstrap.cc:492] T 845da3ee0863430ea3fe9de8c84f2c17 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:30.798738 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_b19a186d285d4b5b845727344f7d155b [id=b28b429fe5844cd6ac6f4de349c9449b]
I20250111 02:57:30.799974 19991 raft_consensus.cc:491] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:30.800470 19991 raft_consensus.cc:513] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.800148 19322 ts_tablet_manager.cc:1397] T 845da3ee0863430ea3fe9de8c84f2c17 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.089s user 0.022s sys 0.047s
I20250111 02:57:30.803771 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a4bc8e9c0c947f6861a01d3e3bd7f31" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:30.804214 19998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 4.278, Last appended by leader: 278, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:30.802721 19991 leader_election.cc:290] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:30.806126 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a4bc8e9c0c947f6861a01d3e3bd7f31" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:30.808645 18841 leader_election.cc:304] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:30.809821 19991 raft_consensus.cc:2743] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:30.804594 19957 raft_consensus.cc:513] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:30.744060 19997 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c34b72b7c70343d3b6f3fae79ce4be6c. Latest consensus state: current_term: 5 leader_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:30.815601 19997 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:30.817539 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0]
I20250111 02:57:30.822263 19322 raft_consensus.cc:357] T 845da3ee0863430ea3fe9de8c84f2c17 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:30.825112 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855dac47872d4c5fbf8f5e38b588c7fe" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:30.825762 19195 raft_consensus.cc:3054] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:30.825830 19321 tablet_bootstrap.cc:492] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:30.825533 19481 consensus_queue.cc:1035] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:30.826503 19321 ts_tablet_manager.cc:1397] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.288s user 0.025s sys 0.000s
I20250111 02:57:30.829118 19321 raft_consensus.cc:357] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.830334 19321 raft_consensus.cc:383] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:30.830678 19321 raft_consensus.cc:738] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:30.830685 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855dac47872d4c5fbf8f5e38b588c7fe" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:30.814268 19998 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 4 FOLLOWER]: Advancing to term 5
W20250111 02:57:30.832644 18989 leader_election.cc:343] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:30.835335 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_dfe42fac0a764052a8d2f6add8882b64 [id=b42908f2a2c14d0a84027cb7a1d775b7]
I20250111 02:57:30.830793 19957 leader_election.cc:290] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:30.839847 19991 raft_consensus.cc:491] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:30.840389 19991 raft_consensus.cc:513] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:30.842564 19991 leader_election.cc:290] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:30.843400 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea78feee64340e39404cba0f1bf04db" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:30.823047 19322 raft_consensus.cc:383] T 845da3ee0863430ea3fe9de8c84f2c17 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:30.847716 19322 raft_consensus.cc:738] T 845da3ee0863430ea3fe9de8c84f2c17 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:30.848361 19322 consensus_queue.cc:260] T 845da3ee0863430ea3fe9de8c84f2c17 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:30.831452 19321 consensus_queue.cc:260] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.853403 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 206) took 1592 ms. Trace:
I20250111 02:57:30.853788 19326 rpcz_store.cc:276] 0111 02:57:29.137894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:29.138114 (+ 220us) service_pool.cc:225] Handling call
0111 02:57:30.207933 (+1069819us) ts_tablet_manager.cc:1145] Deleting tablet 32afc2f158254811836fcf00f7f409ed
0111 02:57:30.207983 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:30.302076 (+ 94093us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:30.596609 (+294533us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:30.730364 (+133755us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1069015}
I20250111 02:57:30.838723 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]
I20250111 02:57:30.869133 19322 ts_tablet_manager.cc:1428] T 845da3ee0863430ea3fe9de8c84f2c17 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.062s user 0.006s sys 0.000s
I20250111 02:57:30.847759 19883 ts_tablet_manager.cc:1905] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:30.873242 19322 tablet_bootstrap.cc:492] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:30.886714 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea78feee64340e39404cba0f1bf04db" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:30.889092 18841 leader_election.cc:304] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:30.890304 19991 raft_consensus.cc:2743] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:30.891391 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_24e5a7f819cc45dbb7efbd7286ad7005 [id=c04c27fb52054de185e1f3075f89a59a]
I20250111 02:57:30.900205 19321 ts_tablet_manager.cc:1428] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.073s user 0.006s sys 0.000s
I20250111 02:57:30.901394 19321 tablet_bootstrap.cc:492] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:30.903286 19195 raft_consensus.cc:2463] T 855dac47872d4c5fbf8f5e38b588c7fe P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:30.908097 19326 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36190 (request call id 207) took 1760 ms. Trace:
I20250111 02:57:30.908097 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 32afc2f158254811836fcf00f7f409ed (table default.Table_2e1ef9ed4749430f848ae883c239e519 [id=a3db149311a04b458696fd724767db06]) successfully deleted
I20250111 02:57:30.909854 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_23a3f1c40c15458e93c2ef79a3c9e9e2 [id=c12add3740174f66996466994ebe9971]
I20250111 02:57:30.908571 19326 rpcz_store.cc:276] 0111 02:57:29.147401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:29.147594 (+ 193us) service_pool.cc:225] Handling call
0111 02:57:30.907782 (+1760188us) ts_tablet_manager.cc:1145] Deleting tablet b8e8c72c99ac4c369957b571d74baffe
0111 02:57:30.907827 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:30.908021 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1759366}
W20250111 02:57:30.910871 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet b8e8c72c99ac4c369957b571d74baffe because the tablet was not found. No further retry: Not found: Tablet not found: b8e8c72c99ac4c369957b571d74baffe
I20250111 02:57:30.895053 19330 tablet_bootstrap.cc:492] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:30.912535 19322 tablet_bootstrap.cc:654] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:30.914934 19860 raft_consensus.cc:491] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:30.915431 19860 raft_consensus.cc:513] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.917618 19860 leader_election.cc:290] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:30.920423 19991 raft_consensus.cc:491] T 845da3ee0863430ea3fe9de8c84f2c17 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:30.920990 19991 raft_consensus.cc:513] T 845da3ee0863430ea3fe9de8c84f2c17 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:30.921656 19330 ts_tablet_manager.cc:1397] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.560s user 0.008s sys 0.003s
I20250111 02:57:30.923257 19991 leader_election.cc:290] T 845da3ee0863430ea3fe9de8c84f2c17 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:30.924837 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845da3ee0863430ea3fe9de8c84f2c17" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:30.924626 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845da3ee0863430ea3fe9de8c84f2c17" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:30.925647 19197 raft_consensus.cc:2388] T 845da3ee0863430ea3fe9de8c84f2c17 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:30.925719 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd364f9a6894d19826deb59a8d8a612" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:30.926728 18841 leader_election.cc:304] T 845da3ee0863430ea3fe9de8c84f2c17 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
W20250111 02:57:30.927376 19133 leader_election.cc:343] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:30.927945 19991 raft_consensus.cc:3054] T 845da3ee0863430ea3fe9de8c84f2c17 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:30.929778 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd364f9a6894d19826deb59a8d8a612" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:30.931579 18986 leader_election.cc:304] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:30.931492 19957 consensus_queue.cc:1035] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:30.932547 19995 raft_consensus.cc:2798] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:30.933024 19995 raft_consensus.cc:695] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:30.933876 19995 consensus_queue.cc:237] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:30.944342 19321 tablet_bootstrap.cc:654] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:30.950260 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_bf5cf724093d4691a60379b87cfab2f0 [id=c151a212f267426a9cb6180c938a5815]
I20250111 02:57:30.943603 19330 raft_consensus.cc:357] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.951977 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_aea3cbca3247424c9a12e310135fe0b9 [id=c151ba8ccc454372b4ad7621b6bc7aeb]
I20250111 02:57:30.953950 19991 raft_consensus.cc:2743] T 845da3ee0863430ea3fe9de8c84f2c17 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:30.954468 19330 raft_consensus.cc:383] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250111 02:57:30.967319 19131 leader_election.cc:343] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:30.967748 19131 leader_election.cc:304] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:30.969415 18901 raft_consensus.cc:3054] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:30.969659 19754 raft_consensus.cc:2743] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:30.976526 19321 tablet_bootstrap.cc:492] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:30.977138 19321 ts_tablet_manager.cc:1397] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.076s user 0.012s sys 0.000s
I20250111 02:57:30.981003 19322 tablet_bootstrap.cc:492] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:30.981473 19322 ts_tablet_manager.cc:1397] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.108s user 0.011s sys 0.000s
I20250111 02:57:30.984019 19322 raft_consensus.cc:357] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:30.984778 19322 raft_consensus.cc:383] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:30.985060 19322 raft_consensus.cc:738] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:30.962807 19883 ts_tablet_manager.cc:1918] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:30.992875 19883 log.cc:1198] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/1b2377893bfc426fa00d5ddacd9c5961
I20250111 02:57:30.991406 19330 raft_consensus.cc:738] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:30.993837 19883 ts_tablet_manager.cc:1939] T 1b2377893bfc426fa00d5ddacd9c5961 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:30.996349 19330 consensus_queue.cc:260] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:30.999372 19322 consensus_queue.cc:260] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:31.001019 19321 raft_consensus.cc:357] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.001822 19321 raft_consensus.cc:383] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:31.002101 19321 raft_consensus.cc:738] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:31.002779 19321 consensus_queue.cc:260] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.005004 19883 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 201) took 3082 ms. Trace:
W20250111 02:57:31.008083 19975 op_tracker.cc:284] OpTracker waiting for 5 outstanding ops to complete now for 1.012s
I20250111 02:57:31.008491 19975 op_tracker.cc:286] Dumping up to 50 currently running ops:
I20250111 02:57:31.008723 19975 op_tracker.cc:290] R-P WriteOp [type=LEADER, start_time=2025-01-11 02:57:29, state=WriteOpState 0x7b58000d2c00 [op_id=(), ts=7112967164652597248, rows=[INSERT (int8 entry_type=<redacted>, string entry_id=<redacted>, string metadata=<redacted>), INSERT (int8 entry_type=<redacted>, string entry_id=<redacted>, string metadata=<redacted>)]]]
I20250111 02:57:31.009739 19975 op_tracker.cc:290] R-P WriteOp [type=LEADER, start_time=2025-01-11 02:57:29, state=WriteOpState 0x7b58000f5d00 [op_id=(), ts=7112967164596736000, rows=[MUTATE (int8 entry_type=<redacted>, string entry_id=<redacted>) SET metadata=<redacted>]]]
I20250111 02:57:31.012250 19322 ts_tablet_manager.cc:1428] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.030s user 0.007s sys 0.001s
I20250111 02:57:31.013286 19322 tablet_bootstrap.cc:492] T 8414653def014ce98bc38d87aac75bae P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:31.010502 19975 op_tracker.cc:290] R-P WriteOp [type=LEADER, start_time=2025-01-11 02:57:29, state=WriteOpState 0x7b5800173d00 [op_id=(), ts=7112967164186255360, rows=[INSERT (int8 entry_type=<redacted>, string entry_id=<redacted>, string metadata=<redacted>), MUTATE (int8 entry_type=<redacted>, string entry_id=<redacted>) SET metadata=<redacted>]]]
I20250111 02:57:31.015369 19975 op_tracker.cc:290] R-P WriteOp [type=LEADER, start_time=2025-01-11 02:57:29, state=WriteOpState 0x7b5800123300 [op_id=(), ts=7112967164921868288, rows=[INSERT (int8 entry_type=<redacted>, string entry_id=<redacted>, string metadata=<redacted>), INSERT (int8 entry_type=<redacted>, string entry_id=<redacted>, string metadata=<redacted>)]]]
I20250111 02:57:31.016023 19975 op_tracker.cc:290] R-P WriteOp [type=LEADER, start_time=2025-01-11 02:57:29, state=WriteOpState 0x7b58000e5b00 [op_id=(), ts=7112967164580909056, rows=[INSERT (int8 entry_type=<redacted>, string entry_id=<redacted>, string metadata=<redacted>), INSERT (int8 entry_type=<redacted>, string entry_id=<redacted>, string metadata=<redacted>)]]]
I20250111 02:57:31.005434 19883 rpcz_store.cc:276] 0111 02:57:27.922389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.925296 (+ 2907us) service_pool.cc:225] Handling call
0111 02:57:30.675005 (+2749709us) ts_tablet_manager.cc:1145] Deleting tablet 1b2377893bfc426fa00d5ddacd9c5961
0111 02:57:30.676094 (+ 1089us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:30.938324 (+262230us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:30.962550 (+ 24226us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:31.004926 (+ 42376us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2687521}
I20250111 02:57:31.024560 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 1b2377893bfc426fa00d5ddacd9c5961 (table default.Table_03f0b101f0c64274a40b43897fe6dcdb [id=d5b01ba62b1a42e7a1dca7769742db0d]) successfully deleted
I20250111 02:57:31.026165 19849 raft_consensus.cc:491] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:31.026396 19883 tablet_replica.cc:331] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:31.026819 19849 raft_consensus.cc:513] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.027405 19883 raft_consensus.cc:2238] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:31.028517 19883 raft_consensus.cc:2267] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
W20250111 02:57:31.029557 18615 leader_election.cc:336] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593): Timed out: RequestConsensusVote RPC to 127.18.20.124:34593 timed out after 0.580s (SENT)
I20250111 02:57:31.030225 19849 leader_election.cc:290] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:31.030673 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd364f9a6894d19826deb59a8d8a612" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:31.032387 18989 leader_election.cc:343] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:31.032568 19321 ts_tablet_manager.cc:1428] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.055s user 0.008s sys 0.000s
I20250111 02:57:31.033596 19321 tablet_bootstrap.cc:492] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:31.036854 19330 ts_tablet_manager.cc:1428] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.110s user 0.006s sys 0.000s
I20250111 02:57:31.039199 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927]
I20250111 02:57:31.040846 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd364f9a6894d19826deb59a8d8a612" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:31.041529 19197 raft_consensus.cc:2463] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:31.046545 19849 consensus_queue.cc:1035] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:31.047427 18986 leader_election.cc:304] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:31.052516 18900 raft_consensus.cc:3054] T 5a4bc8e9c0c947f6861a01d3e3bd7f31 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:31.030953 19883 ts_tablet_manager.cc:1905] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:31.058655 19322 tablet_bootstrap.cc:654] T 8414653def014ce98bc38d87aac75bae P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:31.059532 19754 raft_consensus.cc:491] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:31.060068 19754 raft_consensus.cc:513] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:31.062332 19754 leader_election.cc:290] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:31.064646 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bd42ad66064bcaa043746081f1c759" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:31.066192 19133 leader_election.cc:343] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:31.073719 19683 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 5 FOLLOWER]: Refusing update from remote peer c34b72b7c70343d3b6f3fae79ce4be6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 233. Preceding OpId from leader: term: 5 index: 274. (index mismatch)
I20250111 02:57:31.073999 19321 tablet_bootstrap.cc:654] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:31.075649 19997 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 231, Time since last communication: 0.001s
I20250111 02:57:31.096843 19322 tablet_bootstrap.cc:492] T 8414653def014ce98bc38d87aac75bae P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:31.097318 19322 ts_tablet_manager.cc:1397] T 8414653def014ce98bc38d87aac75bae P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.084s user 0.008s sys 0.002s
I20250111 02:57:31.099843 19322 raft_consensus.cc:357] T 8414653def014ce98bc38d87aac75bae P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:31.100682 19322 raft_consensus.cc:383] T 8414653def014ce98bc38d87aac75bae P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:31.100965 19322 raft_consensus.cc:738] T 8414653def014ce98bc38d87aac75bae P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
W20250111 02:57:31.107610 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 60278.8us
I20250111 02:57:31.107376 19322 consensus_queue.cc:260] T 8414653def014ce98bc38d87aac75bae P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:31.109005 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bd42ad66064bcaa043746081f1c759" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:31.109961 19322 ts_tablet_manager.cc:1428] T 8414653def014ce98bc38d87aac75bae P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
W20250111 02:57:31.110289 19131 leader_election.cc:343] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:31.110702 19131 leader_election.cc:304] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:31.111860 19754 raft_consensus.cc:2743] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:31.112311 19849 raft_consensus.cc:2798] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:31.112740 19849 raft_consensus.cc:491] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:31.112860 19321 tablet_bootstrap.cc:492] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:31.113204 19849 raft_consensus.cc:3054] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:31.113287 19321 ts_tablet_manager.cc:1397] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.080s user 0.011s sys 0.000s
I20250111 02:57:31.113783 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_cce40438ec8e4527b01088af6b7b0981 [id=cb42d3930a9a44bfb5dc40feb979d505]
I20250111 02:57:31.082988 19330 tablet_bootstrap.cc:492] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:31.122145 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5]
I20250111 02:57:31.124131 19321 raft_consensus.cc:357] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.124938 19321 raft_consensus.cc:383] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:31.125221 19321 raft_consensus.cc:738] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:31.125861 19321 consensus_queue.cc:260] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.137663 19321 ts_tablet_manager.cc:1428] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20250111 02:57:31.138726 19321 tablet_bootstrap.cc:492] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:31.139293 19322 tablet_bootstrap.cc:492] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:31.140988 19330 tablet_bootstrap.cc:654] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:31.156940 19322 tablet_bootstrap.cc:654] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:31.146777 19849 raft_consensus.cc:513] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.169226 19883 ts_tablet_manager.cc:1918] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:31.169741 19883 log.cc:1198] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/5fe8aa967fcb4bd7a48f6f0f69bea496
I20250111 02:57:31.170753 19883 ts_tablet_manager.cc:1939] T 5fe8aa967fcb4bd7a48f6f0f69bea496 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:31.173307 19883 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 204) took 3247 ms. Trace:
I20250111 02:57:31.173702 19883 rpcz_store.cc:276] 0111 02:57:27.926366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.926481 (+ 115us) service_pool.cc:225] Handling call
0111 02:57:31.026149 (+3099668us) ts_tablet_manager.cc:1145] Deleting tablet 5fe8aa967fcb4bd7a48f6f0f69bea496
0111 02:57:31.026223 (+ 74us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:31.086240 (+ 60017us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:31.168443 (+ 82203us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:31.173232 (+ 4789us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10,"tablet-delete.queue_time_us":3038849}
I20250111 02:57:31.176496 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd364f9a6894d19826deb59a8d8a612" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:31.177914 18989 leader_election.cc:343] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:31.179010 19849 leader_election.cc:290] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:31.181691 19321 tablet_bootstrap.cc:654] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:31.195278 19883 tablet_replica.cc:331] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:31.196345 19883 raft_consensus.cc:2238] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:31.196954 19883 raft_consensus.cc:2267] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:31.201040 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_d7e00ace61e545129a2e99093dba16db [id=d0e7184505c24fc5b9960e8b6d1c955d]
I20250111 02:57:31.202538 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3]
I20250111 02:57:31.205116 19322 tablet_bootstrap.cc:492] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:31.205597 19322 ts_tablet_manager.cc:1397] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.095s user 0.008s sys 0.002s
I20250111 02:57:31.208206 19322 raft_consensus.cc:357] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.208953 19322 raft_consensus.cc:383] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:31.209249 19322 raft_consensus.cc:738] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:31.209867 19322 consensus_queue.cc:260] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.213351 19322 ts_tablet_manager.cc:1428] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250111 02:57:31.214337 19322 tablet_bootstrap.cc:492] T 2cd364f9a6894d19826deb59a8d8a612 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:31.218369 19330 tablet_bootstrap.cc:492] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:31.224933 19330 ts_tablet_manager.cc:1397] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.142s user 0.007s sys 0.019s
I20250111 02:57:31.228960 19883 ts_tablet_manager.cc:1905] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:31.232261 19757 raft_consensus.cc:491] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:31.232463 19330 raft_consensus.cc:357] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.233155 19330 raft_consensus.cc:383] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:31.232744 19757 raft_consensus.cc:513] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.233400 19330 raft_consensus.cc:738] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:31.233896 19321 tablet_bootstrap.cc:492] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:31.234563 19321 ts_tablet_manager.cc:1397] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.096s user 0.012s sys 0.002s
I20250111 02:57:31.236152 19757 leader_election.cc:290] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:31.236039 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd063f8889b4bcdb9656466f0b144f3" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:31.237480 19133 leader_election.cc:343] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:31.238565 19330 consensus_queue.cc:260] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.237120 19321 raft_consensus.cc:357] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.247262 19321 raft_consensus.cc:383] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:31.247588 19321 raft_consensus.cc:738] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:31.247476 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd364f9a6894d19826deb59a8d8a612" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:31.248219 19192 raft_consensus.cc:3054] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:31.248363 19321 consensus_queue.cc:260] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.253360 19330 ts_tablet_manager.cc:1428] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.028s user 0.007s sys 0.001s
I20250111 02:57:31.255579 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 (table default.Table_f731216a607f4fdaadd2e8dbedf7552d [id=e77c4aa40ab44fb3bc98193e503be4cf]) successfully deleted
I20250111 02:57:31.256512 19330 tablet_bootstrap.cc:492] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
W20250111 02:57:31.257524 18615 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 3fa8530e894c498cbbfa73a3ba1c751e (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3fa8530e894c498cbbfa73a3ba1c751e consensus state has no leader
I20250111 02:57:31.264892 18638 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 5 FOLLOWER]: Refusing update from remote peer c34b72b7c70343d3b6f3fae79ce4be6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 278. Preceding OpId from leader: term: 5 index: 274. (term mismatch)
I20250111 02:57:31.265650 18638 pending_rounds.cc:77] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Aborting all ops after (but not including) 273
I20250111 02:57:31.266227 18638 pending_rounds.cc:99] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Aborting uncommitted WRITE_OP operation due to leader change: 2.274
W20250111 02:57:31.269358 18628 catalog_manager.cc:2177] Aborted: an error occurred while writing to the sys-catalog: Op aborted by new leader
I20250111 02:57:31.272892 18628 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47522 (request call id 262) took 3416 ms. Trace:
I20250111 02:57:31.273133 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd063f8889b4bcdb9656466f0b144f3" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:31.273278 18628 rpcz_store.cc:276] 0111 02:57:27.856904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.929124 (+ 72220us) service_pool.cc:225] Handling call
0111 02:57:28.055423 (+126299us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 02:57:28.169973 (+114550us) catalog_manager.cc:2122] Created new table and tablet info
0111 02:57:31.272757 (+3102784us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:29.163485 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:29.163865 (+ 380us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:29.163888 (+ 23us) write_op.cc:435] Acquired schema lock
0111 02:57:29.163904 (+ 16us) tablet.cc:659] Decoding operations
0111 02:57:29.164210 (+ 306us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:29.164921 (+ 711us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:29.164951 (+ 30us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:29.165170 (+ 219us) tablet.cc:698] Row locks acquired
0111 02:57:29.165184 (+ 14us) write_op.cc:260] PREPARE: finished
0111 02:57:29.165297 (+ 113us) write_op.cc:270] Start()
0111 02:57:29.165382 (+ 85us) write_op.cc:276] Timestamp: P: 1736564249165261 usec, L: 0
0111 02:57:29.165396 (+ 14us) op_driver.cc:348] REPLICATION: starting
0111 02:57:29.165898 (+ 502us) log.cc:844] Serialized 433 byte log entry
0111 02:57:31.267363 (+2101465us) op_driver.cc:464] REPLICATION: finished
0111 02:57:31.267577 (+ 214us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0111 02:57:31.267689 (+ 112us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:31.268122 (+ 433us) write_op.cc:454] Released schema lock
0111 02:57:31.268250 (+ 128us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":789287,"prepare.run_cpu_time_us":1989,"prepare.run_wall_time_us":3603,"replication_time_us":2101453}]]}
I20250111 02:57:31.279143 19321 ts_tablet_manager.cc:1428] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.044s user 0.004s sys 0.004s
I20250111 02:57:31.279564 19322 tablet_bootstrap.cc:654] T 2cd364f9a6894d19826deb59a8d8a612 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:31.279827 19131 leader_election.cc:343] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:31.280263 19131 leader_election.cc:304] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
W20250111 02:57:31.281025 19254 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.20.125:40359)
I20250111 02:57:31.281334 19754 raft_consensus.cc:2743] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:31.276315 18638 pending_rounds.cc:99] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Aborting uncommitted WRITE_OP operation due to leader change: 2.275
W20250111 02:57:31.289244 18628 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.65:43151 (request call id 56) took 1917 ms (client timeout 1000 ms). Trace:
W20250111 02:57:31.290580 18627 catalog_manager.cc:3841] Aborted: an error occurred while updating the sys-catalog: Op aborted by new leader
I20250111 02:57:31.292568 18627 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47522 (request call id 263) took 3215 ms. Trace:
I20250111 02:57:31.292866 18627 rpcz_store.cc:276] 0111 02:57:28.077663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:28.822015 (+744352us) service_pool.cc:225] Handling call
0111 02:57:28.827557 (+ 5542us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d53986 00007fd427ac0644 00007fd427ac05d2 00007fd427ac0501 00007fd427ac047c 00007fd427abf732 00007fd425d79dfc 00007fd425d79236
0111 02:57:28.830095 (+ 2538us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0111 02:57:28.831462 (+ 1367us) catalog_manager.cc:3627] Apply alter schema
0111 02:57:28.842729 (+ 11267us) catalog_manager.cc:3659] Acquired catalog manager lock
0111 02:57:28.842980 (+ 251us) catalog_manager.cc:3817] Updating metadata on disk
0111 02:57:31.292468 (+2449488us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:29.167760 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:29.168535 (+ 775us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:29.168557 (+ 22us) write_op.cc:435] Acquired schema lock
0111 02:57:29.168570 (+ 13us) tablet.cc:659] Decoding operations
0111 02:57:29.168821 (+ 251us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:29.168857 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:29.168878 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:29.169039 (+ 161us) tablet.cc:698] Row locks acquired
0111 02:57:29.169052 (+ 13us) write_op.cc:260] PREPARE: finished
0111 02:57:29.169190 (+ 138us) write_op.cc:270] Start()
0111 02:57:29.169275 (+ 85us) write_op.cc:276] Timestamp: P: 1736564249169125 usec, L: 0
0111 02:57:29.169291 (+ 16us) op_driver.cc:348] REPLICATION: starting
0111 02:57:29.169892 (+ 601us) log.cc:844] Serialized 405 byte log entry
0111 02:57:31.287912 (+2118020us) op_driver.cc:464] REPLICATION: finished
0111 02:57:31.287964 (+ 52us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0111 02:57:31.287988 (+ 24us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:31.289810 (+ 1822us) write_op.cc:454] Released schema lock
0111 02:57:31.289874 (+ 64us) write_op.cc:335] FINISH: Op aborted
Metrics: {"spinlock_wait_cycles":10570496,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":322959,"prepare.run_cpu_time_us":1866,"prepare.run_wall_time_us":3272,"replication_time_us":2118406}]]}
I20250111 02:57:31.296643 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_6823aef149894f49822fe303c036446d [id=d5ffe760744748a98f80f25b8092f9fa]
W20250111 02:57:31.297204 18627 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.67:53171 (request call id 43) took 1092 ms (client timeout 1000 ms). Trace:
W20250111 02:57:31.297571 18627 rpcz_store.cc:269] 0111 02:57:30.204584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:31.297011 (+1092427us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:31.297140 (+ 129us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:31.300266 19254 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.18.20.125:40359)
W20250111 02:57:31.300648 18627 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.66:37049 (request call id 42) took 1029 ms (client timeout 1000 ms). Trace:
W20250111 02:57:31.301074 18627 rpcz_store.cc:269] 0111 02:57:30.269702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:31.298902 (+1029200us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:31.299019 (+ 117us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250111 02:57:31.290910 18638 pending_rounds.cc:99] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Aborting uncommitted WRITE_OP operation due to leader change: 2.276
I20250111 02:57:31.303206 19754 raft_consensus.cc:491] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:31.303737 19754 raft_consensus.cc:513] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.306555 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e06e867266948df8eb35a85c58f578f" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:31.308012 19133 leader_election.cc:343] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:31.309366 18628 rpcz_store.cc:269] 0111 02:57:29.372428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:31.288862 (+1916434us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:31.289151 (+ 289us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250111 02:57:31.309998 19754 leader_election.cc:290] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:31.311276 19991 raft_consensus.cc:491] T 8414653def014ce98bc38d87aac75bae P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:31.311798 19991 raft_consensus.cc:513] T 8414653def014ce98bc38d87aac75bae P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:31.313977 19991 leader_election.cc:290] T 8414653def014ce98bc38d87aac75bae P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:31.310493 19321 tablet_bootstrap.cc:492] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:31.315099 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8414653def014ce98bc38d87aac75bae" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:31.317600 18615 leader_election.cc:336] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593): Timed out: RequestConsensusVote RPC to 127.18.20.124:34593 timed out after 0.594s (SENT)
I20250111 02:57:31.319362 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8414653def014ce98bc38d87aac75bae" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:31.320289 19321 tablet_bootstrap.cc:654] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:31.321004 18841 leader_election.cc:304] T 8414653def014ce98bc38d87aac75bae P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:31.321759 19991 raft_consensus.cc:2743] T 8414653def014ce98bc38d87aac75bae P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:31.322391 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e06e867266948df8eb35a85c58f578f" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:31.324159 18629 catalog_manager.cc:2177] Aborted: an error occurred while writing to the sys-catalog: Op aborted by new leader
W20250111 02:57:31.324224 19131 leader_election.cc:343] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:31.324609 19131 leader_election.cc:304] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:31.325227 18629 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47522 (request call id 264) took 2440 ms. Trace:
I20250111 02:57:31.325578 19754 raft_consensus.cc:2743] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:31.325517 18629 rpcz_store.cc:276] 0111 02:57:28.885202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:29.065849 (+180647us) service_pool.cc:225] Handling call
0111 02:57:29.136465 (+ 70616us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 02:57:29.173935 (+ 37470us) catalog_manager.cc:2122] Created new table and tablet info
0111 02:57:31.325152 (+2151217us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:29.181688 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:29.182102 (+ 414us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:29.182132 (+ 30us) write_op.cc:435] Acquired schema lock
0111 02:57:29.182145 (+ 13us) tablet.cc:659] Decoding operations
0111 02:57:29.182428 (+ 283us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:29.182464 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:29.182486 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:29.182669 (+ 183us) tablet.cc:698] Row locks acquired
0111 02:57:29.182683 (+ 14us) write_op.cc:260] PREPARE: finished
0111 02:57:29.182797 (+ 114us) write_op.cc:270] Start()
0111 02:57:29.182882 (+ 85us) write_op.cc:276] Timestamp: P: 1736564249182763 usec, L: 0
0111 02:57:29.182896 (+ 14us) op_driver.cc:348] REPLICATION: starting
0111 02:57:29.183449 (+ 553us) log.cc:844] Serialized 433 byte log entry
0111 02:57:31.302411 (+2118962us) op_driver.cc:464] REPLICATION: finished
0111 02:57:31.302466 (+ 55us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0111 02:57:31.302493 (+ 27us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:31.302720 (+ 227us) write_op.cc:454] Released schema lock
0111 02:57:31.302776 (+ 56us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":6205,"prepare.run_cpu_time_us":1984,"prepare.run_wall_time_us":3111,"replication_time_us":2119276}]]}
I20250111 02:57:31.332798 18638 pending_rounds.cc:99] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Aborting uncommitted WRITE_OP operation due to leader change: 2.277
W20250111 02:57:31.337131 18621 catalog_manager.cc:2177] Aborted: an error occurred while writing to the sys-catalog: Op aborted by new leader
I20250111 02:57:31.338165 18621 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47522 (request call id 265) took 2321 ms. Trace:
I20250111 02:57:31.338449 18621 rpcz_store.cc:276] 0111 02:57:29.016981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:29.222156 (+205175us) service_pool.cc:225] Handling call
0111 02:57:29.232747 (+ 10591us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 02:57:29.234261 (+ 1514us) catalog_manager.cc:2122] Created new table and tablet info
0111 02:57:31.338079 (+2103818us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:29.247377 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:29.247771 (+ 394us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:29.247796 (+ 25us) write_op.cc:435] Acquired schema lock
0111 02:57:29.247811 (+ 15us) tablet.cc:659] Decoding operations
0111 02:57:29.248129 (+ 318us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:29.248168 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:29.248192 (+ 24us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:29.248402 (+ 210us) tablet.cc:698] Row locks acquired
0111 02:57:29.248417 (+ 15us) write_op.cc:260] PREPARE: finished
0111 02:57:29.248541 (+ 124us) write_op.cc:270] Start()
0111 02:57:29.248627 (+ 86us) write_op.cc:276] Timestamp: P: 1736564249248503 usec, L: 0
0111 02:57:29.248642 (+ 15us) op_driver.cc:348] REPLICATION: starting
0111 02:57:29.249184 (+ 542us) log.cc:844] Serialized 433 byte log entry
0111 02:57:31.335618 (+2086434us) op_driver.cc:464] REPLICATION: finished
0111 02:57:31.335679 (+ 61us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0111 02:57:31.335716 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:31.335919 (+ 203us) write_op.cc:454] Released schema lock
0111 02:57:31.335961 (+ 42us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":11909,"prepare.run_cpu_time_us":2027,"prepare.run_wall_time_us":2027,"replication_time_us":2086727}]]}
I20250111 02:57:31.344738 18638 pending_rounds.cc:99] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Aborting uncommitted NO_OP operation due to leader change: 4.278
I20250111 02:57:31.344779 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_5e21879e67544c71af627f795fd6d940 [id=d6c0153a97cd42d49f534f48fc5b8287]
I20250111 02:57:31.346124 18638 raft_consensus.cc:2883] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250111 02:57:31.349746 19997 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 272, Time since last communication: 0.000s
I20250111 02:57:31.350406 19997 consensus_queue.cc:1225] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [LEADER]: Peer 8784b9603f8544b79373158aace063af log is divergent from this leader: its last log entry 2.273 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 272
I20250111 02:57:31.353469 18639 pending_rounds.cc:77] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Aborting all ops after (but not including) 272
I20250111 02:57:31.353787 18639 pending_rounds.cc:99] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Aborting uncommitted WRITE_OP operation due to leader change: 2.273
W20250111 02:57:31.355512 18623 catalog_manager.cc:6561] Aborted: an error occurred while writing to the sys-catalog: Op aborted by new leader
I20250111 02:57:31.356477 18623 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47522 (request call id 256) took 3981 ms. Trace:
I20250111 02:57:31.356760 18623 rpcz_store.cc:276] 0111 02:57:27.375036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.613837 (+238801us) service_pool.cc:225] Handling call
0111 02:57:31.356358 (+3742521us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:29.067866 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:29.068222 (+ 356us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:29.068251 (+ 29us) write_op.cc:435] Acquired schema lock
0111 02:57:29.068265 (+ 14us) tablet.cc:659] Decoding operations
0111 02:57:29.068577 (+ 312us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:29.068610 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:29.068631 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:29.068813 (+ 182us) tablet.cc:698] Row locks acquired
0111 02:57:29.068825 (+ 12us) write_op.cc:260] PREPARE: finished
0111 02:57:29.068931 (+ 106us) write_op.cc:270] Start()
0111 02:57:29.069013 (+ 82us) write_op.cc:276] Timestamp: P: 1736564249068910 usec, L: 0
0111 02:57:29.069029 (+ 16us) op_driver.cc:348] REPLICATION: starting
0111 02:57:29.159630 (+ 90601us) log.cc:844] Serialized 625 byte log entry
0111 02:57:29.161887 (+ 2257us) spinlock_profiling.cc:243] Waited 90 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f316d1 00007fd427fa8198 00007fd4284143b1 00007fd428413b88 00007fd428418821 00007fd4284187d9 00007fd428418769 00007fd428418731 00007fd42841795d 000000000059ea04 000000000059e839
0111 02:57:31.354195 (+2192308us) op_driver.cc:464] REPLICATION: finished
0111 02:57:31.354252 (+ 57us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0111 02:57:31.354279 (+ 27us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:31.354355 (+ 76us) write_op.cc:454] Released schema lock
0111 02:57:31.354405 (+ 50us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":673501,"prepare.run_cpu_time_us":3985,"prepare.run_wall_time_us":310064,"raft.queue_time_us":102,"raft.run_cpu_time_us":682,"raft.run_wall_time_us":683,"replication_time_us":2284977,"spinlock_wait_cycles":207106944}]]}
I20250111 02:57:31.364596 18623 master_service.cc:946] ReplaceTablet: received request to replace tablet 1ef92ac716824107916a1ea4fd49d5aa from {username='slave'} at 127.0.0.1:47522
W20250111 02:57:31.379665 19254 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.18.20.125:40359)
I20250111 02:57:31.426827 19754 raft_consensus.cc:491] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:31.431995 19883 ts_tablet_manager.cc:1918] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:31.433118 19754 raft_consensus.cc:513] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.433681 19883 log.cc:1198] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/5cf1ce65ccf54358b95665b9764fbdb0
I20250111 02:57:31.434551 19883 ts_tablet_manager.cc:1939] T 5cf1ce65ccf54358b95665b9764fbdb0 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:31.435099 19754 leader_election.cc:290] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:31.437292 19883 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 206) took 2510 ms. Trace:
I20250111 02:57:31.437716 19883 rpcz_store.cc:276] 0111 02:57:28.926466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:28.926721 (+ 255us) service_pool.cc:225] Handling call
0111 02:57:31.175005 (+2248284us) ts_tablet_manager.cc:1145] Deleting tablet 5cf1ce65ccf54358b95665b9764fbdb0
0111 02:57:31.195114 (+ 20109us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:31.277594 (+ 82480us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:31.427557 (+149963us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:31.437223 (+ 9666us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":95,"tablet-delete.queue_time_us":2247513}
I20250111 02:57:31.427201 19192 raft_consensus.cc:2463] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:31.439520 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef86e1fd412484db83e7d39c9059a90" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:31.440893 19133 leader_election.cc:343] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:31.441028 19883 tablet_replica.cc:331] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:31.441692 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 5cf1ce65ccf54358b95665b9764fbdb0 (table default.Table_61c40191c611402da3829f2ef5379f61 [id=b4790f95b19748c7a1130504d33b0319]) successfully deleted
I20250111 02:57:31.442071 19883 raft_consensus.cc:2238] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:31.442646 19883 raft_consensus.cc:2267] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:31.444679 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_813d4e8bb0f34a46bf96ed3bee6ca86f [id=d790288a097a4416b68b93db6ac847fd]
I20250111 02:57:31.450336 19883 ts_tablet_manager.cc:1905] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:31.465512 19321 tablet_bootstrap.cc:492] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:31.465991 19321 ts_tablet_manager.cc:1397] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.156s user 0.012s sys 0.006s
I20250111 02:57:31.467365 19998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: New leader c34b72b7c70343d3b6f3fae79ce4be6c. Latest consensus state: current_term: 5 leader_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:31.468724 19998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:31.469128 18623 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47522 (request call id 269) took 1838 ms. Trace:
I20250111 02:57:31.469512 18623 rpcz_store.cc:276] 0111 02:57:29.630426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:31.364433 (+1734007us) service_pool.cc:225] Handling call
0111 02:57:31.467901 (+103468us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d55d31 00007fd427ad7d47 00007fd427ad7cd2 00007fd427ad7c01 00007fd427ad7b7c 00007fd427ad6e32 00007fd425d79dfc 00007fd425d79236
0111 02:57:31.469045 (+ 1144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":234736512}
I20250111 02:57:31.471400 19321 raft_consensus.cc:357] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.472147 19321 raft_consensus.cc:383] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:31.472147 18621 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47522 (request call id 268) took 2060 ms. Trace:
I20250111 02:57:31.472410 19321 raft_consensus.cc:738] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:31.472493 18621 rpcz_store.cc:276] 0111 02:57:29.412380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:31.341178 (+1928798us) service_pool.cc:225] Handling call
0111 02:57:31.471373 (+130195us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d53306 00007fd427ab7044 00007fd427ab6fd2 00007fd427ab6f01 00007fd427ab6e7c 00007fd427ab6132 00007fd425d79dfc 00007fd425d79236
0111 02:57:31.472026 (+ 653us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":279204224}
I20250111 02:57:31.473060 19321 consensus_queue.cc:260] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.477686 18629 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47522 (request call id 266) took 2452 ms. Trace:
I20250111 02:57:31.478148 18629 rpcz_store.cc:276] 0111 02:57:29.024776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:31.330402 (+2305626us) service_pool.cc:225] Handling call
0111 02:57:31.476730 (+146328us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d53616 00007fd427abbb44 00007fd427abbad2 00007fd427abba01 00007fd427abb97c 00007fd427abac32 00007fd425d79dfc 00007fd425d79236
0111 02:57:31.477598 (+ 868us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":300185472}
I20250111 02:57:31.483232 19322 tablet_bootstrap.cc:492] T 2cd364f9a6894d19826deb59a8d8a612 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:31.483700 19322 ts_tablet_manager.cc:1397] T 2cd364f9a6894d19826deb59a8d8a612 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.270s user 0.020s sys 0.018s
I20250111 02:57:31.485173 18628 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47522 (request call id 267) took 2132 ms. Trace:
I20250111 02:57:31.485594 18628 rpcz_store.cc:276] 0111 02:57:29.352560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:31.333471 (+1980911us) service_pool.cc:225] Handling call
0111 02:57:31.480995 (+147524us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d53986 00007fd427ac0644 00007fd427ac05d2 00007fd427ac0501 00007fd427ac047c 00007fd427abf732 00007fd425d79dfc 00007fd425d79236
0111 02:57:31.485094 (+ 4099us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":305524480}
I20250111 02:57:31.486238 19322 raft_consensus.cc:357] T 2cd364f9a6894d19826deb59a8d8a612 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.488411 19321 ts_tablet_manager.cc:1428] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20250111 02:57:31.492131 19849 consensus_queue.cc:1035] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:31.493537 18986 leader_election.cc:304] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:31.494282 19849 raft_consensus.cc:2798] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:31.494707 19849 raft_consensus.cc:695] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:31.495993 19849 consensus_queue.cc:237] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.486984 19322 raft_consensus.cc:383] T 2cd364f9a6894d19826deb59a8d8a612 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:31.503407 19322 raft_consensus.cc:738] T 2cd364f9a6894d19826deb59a8d8a612 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:31.504132 19322 consensus_queue.cc:260] T 2cd364f9a6894d19826deb59a8d8a612 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:31.507874 19254 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.18.20.125:40359)
I20250111 02:57:31.511786 19321 tablet_bootstrap.cc:492] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:31.515997 19957 consensus_queue.cc:1035] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:31.516968 19322 ts_tablet_manager.cc:1428] T 2cd364f9a6894d19826deb59a8d8a612 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20250111 02:57:31.517956 19322 tablet_bootstrap.cc:492] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
W20250111 02:57:31.526768 18627 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.65:43151 (request call id 57) took 1130 ms (client timeout 1000 ms). Trace:
I20250111 02:57:31.529227 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef86e1fd412484db83e7d39c9059a90" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:31.533857 19321 tablet_bootstrap.cc:654] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:31.536628 19330 tablet_bootstrap.cc:654] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:31.539331 18627 rpcz_store.cc:269] 0111 02:57:30.381862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:31.310584 (+928722us) service_pool.cc:225] Handling call
0111 02:57:31.497310 (+186726us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d51dee 00007fd427aaffc4 00007fd427aaff52 00007fd427aafe81 00007fd427aafdfc 00007fd427aaf0b2 00007fd425d79dfc 00007fd425d79236
0111 02:57:31.512511 (+ 15201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":265605888}
I20250111 02:57:31.542021 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_23d632d4476c46b78d27619201589641 [id=db5ffbbe4fe7424f943fbaca17103d84]
W20250111 02:57:31.545645 19131 leader_election.cc:343] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:31.546098 19131 leader_election.cc:304] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:31.547322 20027 raft_consensus.cc:2743] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:31.558854 19322 tablet_bootstrap.cc:654] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:31.531169 18629 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:55106 (request call id 2) took 2324 ms. Trace:
I20250111 02:57:31.561357 18629 rpcz_store.cc:276] 0111 02:57:29.182230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:31.499746 (+2317516us) service_pool.cc:225] Handling call
0111 02:57:31.506130 (+ 6384us) spinlock_profiling.cc:243] Waited 5.02 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d54dea 00007fd427ae5e47 00007fd427ae5dd2 00007fd427ae5d01 00007fd427ae5c7c 00007fd427ae4f32 00007fd425d79dfc 00007fd425d79236
0111 02:57:31.506978 (+ 848us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11535616}
I20250111 02:57:31.568523 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_ea068446b5434b3ca5f0afa8386c0655 [id=dc09e842add34ef099edfd1296255c4b]
I20250111 02:57:31.580549 19883 ts_tablet_manager.cc:1918] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:31.581068 19883 log.cc:1198] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/4b302fcba27244c193586bc25fc9305f
I20250111 02:57:31.581995 19883 ts_tablet_manager.cc:1939] T 4b302fcba27244c193586bc25fc9305f P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:31.587442 19975 catalog_manager.cc:1423] Term changed from 4 to 5 while waiting for master leader catchup. Not loading sys catalog metadata
I20250111 02:57:31.606959 19322 tablet_bootstrap.cc:492] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:31.607575 19322 ts_tablet_manager.cc:1397] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.090s user 0.007s sys 0.003s
I20250111 02:57:31.610316 19322 raft_consensus.cc:357] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.611014 19322 raft_consensus.cc:383] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:31.611320 19322 raft_consensus.cc:738] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:31.611928 19322 consensus_queue.cc:260] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.627270 20020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c34b72b7c70343d3b6f3fae79ce4be6c. Latest consensus state: current_term: 5 leader_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:31.628129 20020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:31.629575 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_a7b2b5af5dfb439a84159b75de0d03b1 [id=dc7854580f744d4083f6aa3f0f0f7bf2]
I20250111 02:57:31.647607 20020 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c34b72b7c70343d3b6f3fae79ce4be6c)
I20250111 02:57:31.648125 20020 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:31.651146 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 6 candidate_status { last_received { term: 5 index: 274 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:31.656268 20020 leader_election.cc:290] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8784b9603f8544b79373158aace063af (127.18.20.125:40359), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:31.660261 19330 tablet_bootstrap.cc:492] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:31.660792 19330 ts_tablet_manager.cc:1397] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.404s user 0.011s sys 0.001s
I20250111 02:57:31.663925 19883 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32868 (request call id 207) took 2736 ms. Trace:
I20250111 02:57:31.664321 19883 rpcz_store.cc:276] 0111 02:57:28.927955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:28.928101 (+ 146us) service_pool.cc:225] Handling call
0111 02:57:31.438962 (+2510861us) ts_tablet_manager.cc:1145] Deleting tablet 4b302fcba27244c193586bc25fc9305f
0111 02:57:31.439044 (+ 82us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0111 02:57:31.506983 (+ 67939us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:31.551015 (+ 44032us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:31.663823 (+112808us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2510272}
I20250111 02:57:31.665772 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 4b302fcba27244c193586bc25fc9305f (table default.Table_8292a1a0b58247ca951f8f8a951a83ad [id=d7462be28a2c445889cf333c56ae28a5]) successfully deleted
I20250111 02:57:31.669116 19321 tablet_bootstrap.cc:492] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:31.669584 19321 ts_tablet_manager.cc:1397] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.158s user 0.007s sys 0.005s
I20250111 02:57:31.673555 19321 raft_consensus.cc:357] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.674348 19321 raft_consensus.cc:383] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:31.674620 19321 raft_consensus.cc:738] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:31.675751 19321 consensus_queue.cc:260] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.679729 18900 raft_consensus.cc:3054] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:31.680562 19321 ts_tablet_manager.cc:1428] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20250111 02:57:31.686829 19321 tablet_bootstrap.cc:492] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
W20250111 02:57:31.691833 18989 consensus_peers.cc:487] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:31.699187 18901 raft_consensus.cc:3054] T 8414653def014ce98bc38d87aac75bae P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:31.723809 19849 consensus_queue.cc:1035] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:31.730850 19330 raft_consensus.cc:357] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.730593 19321 tablet_bootstrap.cc:654] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:31.731638 19997 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:31.732674 19997 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:31.734894 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_f9181ca352384593b93119fb3e0cd812 [id=e0ac9d601c464a3daaaa63c4cb90e471]
I20250111 02:57:31.735877 19322 ts_tablet_manager.cc:1428] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.128s user 0.006s sys 0.001s
I20250111 02:57:31.755983 19330 raft_consensus.cc:383] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:31.756352 19330 raft_consensus.cc:738] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:31.756997 19330 consensus_queue.cc:260] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.781323 19322 tablet_bootstrap.cc:492] T 855dac47872d4c5fbf8f5e38b588c7fe P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:31.789865 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_c4faffc9659848ea9bff581e8d88d40e [id=e709d8f708144de8a18d241f0e0306a0]
I20250111 02:57:31.794343 20046 raft_consensus.cc:491] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:31.797206 20046 raft_consensus.cc:513] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.801050 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd063f8889b4bcdb9656466f0b144f3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:31.803673 18989 leader_election.cc:343] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:31.807276 19849 raft_consensus.cc:491] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:31.808050 19849 raft_consensus.cc:513] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.810973 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c6b166a29943ceb15f093fac728c79" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:31.812572 18989 leader_election.cc:343] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:31.813287 19991 raft_consensus.cc:491] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:31.813767 19991 raft_consensus.cc:513] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.815781 19991 leader_election.cc:290] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:31.817865 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf1ce65ccf54358b95665b9764fbdb0" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:31.821751 19998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:31.822630 19998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:31.824050 20044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:31.840782 19330 ts_tablet_manager.cc:1428] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.180s user 0.005s sys 0.001s
I20250111 02:57:31.845252 20044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:31.846360 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_799d0b8328de40a7bacb58ea433fec72 [id=e9efdc6c94574ddca092dd6b9f3b8419]
I20250111 02:57:31.850770 19330 tablet_bootstrap.cc:492] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
W20250111 02:57:31.887979 18841 leader_election.cc:343] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Not found: Tablet not found: 5cf1ce65ccf54358b95665b9764fbdb0
I20250111 02:57:31.888407 18841 leader_election.cc:304] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:31.889587 19991 raft_consensus.cc:2743] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:31.886920 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd063f8889b4bcdb9656466f0b144f3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:31.895375 19194 raft_consensus.cc:2463] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
W20250111 02:57:31.902537 18986 consensus_peers.cc:487] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cf1ce65ccf54358b95665b9764fbdb0. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:31.903374 18986 leader_election.cc:304] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:31.904850 19957 raft_consensus.cc:2798] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:31.906843 19957 raft_consensus.cc:491] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:31.908213 19957 raft_consensus.cc:3054] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:31.927778 19991 raft_consensus.cc:491] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:31.928364 19991 raft_consensus.cc:513] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.929670 18906 raft_consensus.cc:3054] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:31.942528 19991 leader_election.cc:290] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:31.949306 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b302fcba27244c193586bc25fc9305f" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:31.937593 20046 leader_election.cc:290] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:31.971393 19991 raft_consensus.cc:491] T 2cd364f9a6894d19826deb59a8d8a612 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:31.972105 19991 raft_consensus.cc:513] T 2cd364f9a6894d19826deb59a8d8a612 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:31.976665 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd364f9a6894d19826deb59a8d8a612" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:31.979053 19991 leader_election.cc:290] T 2cd364f9a6894d19826deb59a8d8a612 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:31.980350 19849 leader_election.cc:290] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:31.991791 18841 leader_election.cc:343] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Not found: Tablet not found: 4b302fcba27244c193586bc25fc9305f
I20250111 02:57:31.992246 18841 leader_election.cc:304] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:31.998564 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c6b166a29943ceb15f093fac728c79" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:31.999429 19194 raft_consensus.cc:2463] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:31.999672 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd364f9a6894d19826deb59a8d8a612" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:32.001679 19196 raft_consensus.cc:2388] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:32.002964 18841 leader_election.cc:304] T 2cd364f9a6894d19826deb59a8d8a612 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:32.007489 18986 leader_election.cc:304] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:32.009248 20055 raft_consensus.cc:2798] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:32.009584 20055 raft_consensus.cc:491] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:32.010026 20055 raft_consensus.cc:3054] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:32.010337 19849 consensus_queue.cc:1035] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:32.016530 20058 raft_consensus.cc:3054] T 2cd364f9a6894d19826deb59a8d8a612 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:32.033771 19322 tablet_bootstrap.cc:654] T 855dac47872d4c5fbf8f5e38b588c7fe P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:32.036387 18989 consensus_peers.cc:487] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:32.039783 19991 raft_consensus.cc:2743] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:32.047549 20055 raft_consensus.cc:513] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.056860 20043 raft_consensus.cc:491] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 54628fbcec2f4359aa9e26a734d4d29f)
I20250111 02:57:32.057404 20043 raft_consensus.cc:513] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:32.060425 20043 leader_election.cc:290] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:32.064728 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a03880a8b0e46b7858ed1c2c2dddcaa" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:32.087235 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_902d917b19634ad5b20fadd455c0ba88 [id=ec311d46e9224fd38d1864b43717da4a]
I20250111 02:57:32.090579 20044 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c34b72b7c70343d3b6f3fae79ce4be6c)
I20250111 02:57:32.091465 20044 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:32.095654 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 6 candidate_status { last_received { term: 5 index: 274 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:32.096314 19683 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 5.
I20250111 02:57:32.097260 20044 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:32.097836 18616 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af; no voters:
I20250111 02:57:32.098788 20044 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250111 02:57:32.099298 20044 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 5 FOLLOWER]: Starting leader election (detected failure of leader c34b72b7c70343d3b6f3fae79ce4be6c)
I20250111 02:57:32.099704 20058 raft_consensus.cc:2743] T 2cd364f9a6894d19826deb59a8d8a612 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:32.099856 20044 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 5 FOLLOWER]: Advancing to term 6
I20250111 02:57:32.102473 19860 raft_consensus.cc:491] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:32.103044 19860 raft_consensus.cc:513] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.126549 19860 leader_election.cc:290] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:32.126397 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f11541b436e42b7b0728da2aaec48f7" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:32.127420 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_b44f1ebbaf7045e2bcfefcee1c91c0d9 [id=ec5cf5f9b6234008914c0ed664260dcb]
W20250111 02:57:32.128239 19133 leader_election.cc:343] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:32.148274 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_e091dc5f47d9492aa86ff883e04fe415 [id=ee752760552944289c461f1c4c50d90f]
I20250111 02:57:32.061216 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a03880a8b0e46b7858ed1c2c2dddcaa" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:32.149775 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_49e4dd58d29f4cb6a0ed4f85138713b8 [id=f1373922978a4598b888e1863525942e]
I20250111 02:57:32.151391 18842 leader_election.cc:304] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:32.151562 20055 leader_election.cc:290] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:32.152411 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c6b166a29943ceb15f093fac728c79" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:32.152631 20058 raft_consensus.cc:2743] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:32.153038 19196 raft_consensus.cc:3054] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:32.156873 19330 tablet_bootstrap.cc:654] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:32.141580 19130 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 60076.2us
I20250111 02:57:32.171319 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c6b166a29943ceb15f093fac728c79" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:32.171721 18625 catalog_manager.cc:2786] Illegal state: an error occurred while updating the sys-catalog: replica 8784b9603f8544b79373158aace063af is not leader of this config: current role FOLLOWER
W20250111 02:57:32.173079 18989 leader_election.cc:343] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:32.172312 18625 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47522 (request call id 255) took 4844 ms. Trace:
I20250111 02:57:32.173806 18625 rpcz_store.cc:276] 0111 02:57:27.328072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.606469 (+278397us) service_pool.cc:225] Handling call
0111 02:57:27.608227 (+ 1758us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0111 02:57:27.608780 (+ 553us) catalog_manager.cc:2753] Modifying in-memory table state
0111 02:57:27.608906 (+ 126us) catalog_manager.cc:2762] Locking tablets
0111 02:57:32.168232 (+4559326us) catalog_manager.cc:2776] Removing table and tablets from system table
0111 02:57:32.171329 (+ 3097us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:32.171349 (+ 20us) write_op.cc:454] Released schema lock
0111 02:57:32.172244 (+ 895us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:32.170901 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 8784b9603f8544b79373158aace063af is not leader of this config: current role FOLLOWER)
0111 02:57:32.170941 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:32.170962 (+ 21us) write_op.cc:454] Released schema lock
0111 02:57:32.170985 (+ 23us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{}]]}
W20250111 02:57:32.175042 20052 log.cc:926] Time spent T 6a03880a8b0e46b7858ed1c2c2dddcaa P 8221604733fe4fc0a1fc32de8d6af71c: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20250111 02:57:32.178093 19322 tablet_bootstrap.cc:492] T 855dac47872d4c5fbf8f5e38b588c7fe P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:32.178552 19322 ts_tablet_manager.cc:1397] T 855dac47872d4c5fbf8f5e38b588c7fe P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.397s user 0.013s sys 0.004s
I20250111 02:57:32.181281 19322 raft_consensus.cc:357] T 855dac47872d4c5fbf8f5e38b588c7fe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:32.182046 19322 raft_consensus.cc:383] T 855dac47872d4c5fbf8f5e38b588c7fe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:32.182340 19322 raft_consensus.cc:738] T 855dac47872d4c5fbf8f5e38b588c7fe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:32.193776 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet 9192160e931445f1acd207604192f759 (table default.Table_bccba04b48cc408d845624ee7ed98e5f [id=1496848ec9594cad889312978436d9c4]). Expected version 1 got 0
I20250111 02:57:32.209434 19321 tablet_bootstrap.cc:492] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:32.209939 19321 ts_tablet_manager.cc:1397] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.525s user 0.022s sys 0.024s
I20250111 02:57:32.217288 19321 raft_consensus.cc:357] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.218144 19321 raft_consensus.cc:383] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:32.218441 19321 raft_consensus.cc:738] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:32.225409 19196 raft_consensus.cc:2463] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:32.227033 18986 leader_election.cc:304] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
W20250111 02:57:32.227339 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 454555us
I20250111 02:57:32.229122 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f11541b436e42b7b0728da2aaec48f7" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:32.231989 20055 raft_consensus.cc:2798] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:32.232456 20055 raft_consensus.cc:695] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:32.233239 20055 consensus_queue.cc:237] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.236006 19955 catalog_manager.cc:670] Loaded metadata for table default.Table_7ee7d20cfb7e4985bdd8539d15e354fe [id=ff66cd83751a45a09d8012cc8a9f1758]
I20250111 02:57:32.230257 18630 catalog_manager.cc:5526] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:32.236809 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet cc7158a7fc624fa4bc816ac3721c610d (table default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605]). Expected version 2 got 0
I20250111 02:57:32.237537 19754 raft_consensus.cc:491] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:32.238099 19754 raft_consensus.cc:513] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.239131 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]). Expected version 2 got 0
I20250111 02:57:32.239766 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet a983a670c2104398ac1a159ac3a1da7c (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19]). Expected version 4 got 0
I20250111 02:57:32.240151 19322 consensus_queue.cc:260] T 855dac47872d4c5fbf8f5e38b588c7fe P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:32.240708 19754 leader_election.cc:290] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:32.242513 19322 ts_tablet_manager.cc:1428] T 855dac47872d4c5fbf8f5e38b588c7fe P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.064s user 0.006s sys 0.000s
W20250111 02:57:32.243862 19131 leader_election.cc:343] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:32.243721 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3052e9707de14e1f9e38236d56908ed3" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:32.244424 19131 leader_election.cc:304] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:32.245249 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3052e9707de14e1f9e38236d56908ed3" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:32.246630 19133 leader_election.cc:343] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:32.256192 19754 raft_consensus.cc:2743] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:32.260998 19321 consensus_queue.cc:260] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:32.258373 19131 leader_election.cc:343] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:32.263831 20044 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:32.264369 19131 leader_election.cc:304] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:32.265887 19754 raft_consensus.cc:2743] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:32.266968 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 6 candidate_status { last_received { term: 5 index: 274 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727"
I20250111 02:57:32.267575 19683 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 5 FOLLOWER]: Advancing to term 6
I20250111 02:57:32.268245 19322 tablet_bootstrap.cc:492] T 0158a8849bf44a4fbb5daca22883478f P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
W20250111 02:57:32.287010 18986 consensus_peers.cc:487] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b302fcba27244c193586bc25fc9305f. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:32.287986 19683 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 6.
I20250111 02:57:32.289335 18616 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af; no voters:
I20250111 02:57:32.290339 19998 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 6 FOLLOWER]: Leader election won for term 6
I20250111 02:57:32.294876 19330 tablet_bootstrap.cc:492] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:32.295322 19330 ts_tablet_manager.cc:1397] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.445s user 0.018s sys 0.002s
I20250111 02:57:32.298233 19330 raft_consensus.cc:357] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.298960 19330 raft_consensus.cc:383] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:32.299247 19330 raft_consensus.cc:738] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
E20250111 02:57:32.300184 18813 catalog_manager.cc:817] Error processing pending assignments: Illegal state: error persisting updated tablet metadata: replica 8784b9603f8544b79373158aace063af is not leader of this config: current role FOLLOWER
I20250111 02:57:32.299916 19330 consensus_queue.cc:260] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:32.300629 19595 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.124:34593, user_credentials={real_user=slave}} blocked reactor thread for 123956us
I20250111 02:57:32.328761 19957 raft_consensus.cc:513] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.332827 20044 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 6 election: Requested vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:32.346408 19330 ts_tablet_manager.cc:1428] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.051s user 0.006s sys 0.000s
I20250111 02:57:32.347422 19330 tablet_bootstrap.cc:492] T e46894710a72407993b36801f3b90eb0 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:32.350189 19998 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 6 LEADER]: Becoming Leader. State: Replica: 8784b9603f8544b79373158aace063af, State: Running, Role: LEADER
I20250111 02:57:32.353109 20055 consensus_queue.cc:1035] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:32.353204 19998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 274, Committed index: 274, Last appended: 5.274, Last appended by leader: 274, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
E20250111 02:57:32.343525 18630 catalog_manager.cc:5681] Error updating tablets from 54628fbcec2f4359aa9e26a734d4d29f: Service unavailable: leader is not yet ready. Tablet report was: is_incremental: true updated_tablets { tablet_id: "f778763e59a94969b3f62be87e2fd867" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "49616e4e7cc647509e72cde57b87d6fa" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "b5148fa2e6504561bd1b1a19e48993d7" state: RUNNING consensus_state { current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27063 live_row_count: 0 } } updated_tablets { tablet_id: "5a4bc8e9c0c947f6861a01d3e3bd7f31" state: RUNNING consensus_state { current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "f1802e7349934fbb9d5f7eda37c3df20" state: RUNNING consensus_state { current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } } } schema_version: 4 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27063 live_row_count: 0 } } updated_tablets { tablet_id: "b8e8c72c99ac4c369957b571d74baffe" state: STOPPED consensus_state { current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_DELETED } updated_tablets { tablet_id: "5fe8aa967fcb4bd7a48f6f0f69bea496" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "9d6de202f4e547a0b8b90cfa09ad319a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "40fdec982cf643c68ef90798f4f65ad4" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a983a670c2104398ac1a159ac3a1da7c" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "d8ef3f45e5784a29b3b32cc0de6bac84" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "199ca8692f39445f93d8c27e2134a32c" state: RUNNING consensus_state { current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26732 live_row_count: 0 } } updated_tablets { tablet_id: "cc7158a7fc624fa4bc816ac3721c610d" state: RUNNING consensus_state { current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "32afc2f158254811836fcf00f7f409ed" state: RUNNING consensus_state { current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "9192160e931445f1acd207604192f759" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "74592f80afab4513a9eb2eac351c4de5" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } removed_tablet_ids: "29561e984e6d40fea032075f0531dd1f" removed_tablet_ids: "1b2377893bfc426fa00d5ddacd9c5961" removed_tablet_ids: "9fe4e60db4244a9a9696153b630c575b" removed_tablet_ids: "5118d4b9cfbb49e8af6af56180858336" removed_tablet_ids: "ef70b417568049f0b496f4d8e5f7e0e4" removed_tablet_ids: "c28958a1018344a48d18e6c0f03cd0a3" removed_tablet_ids: "d8128b3941d54ce2a72bf2ae762a774f" removed_tablet_ids: "0267a92995ea460786dc26813b5895cf" removed_tablet_ids: "b54311f4269d4575b8998d7f545000e7" sequence_number: 39
I20250111 02:57:32.375245 19321 ts_tablet_manager.cc:1428] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.165s user 0.012s sys 0.000s
I20250111 02:57:32.385188 19321 tablet_bootstrap.cc:492] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:32.388513 18622 catalog_manager.cc:5526] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } }
I20250111 02:57:32.397096 19957 leader_election.cc:290] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:32.398979 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd063f8889b4bcdb9656466f0b144f3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:32.399749 19196 raft_consensus.cc:3054] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:32.399554 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd063f8889b4bcdb9656466f0b144f3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:32.401491 18989 leader_election.cc:343] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:32.407514 19322 tablet_bootstrap.cc:654] T 0158a8849bf44a4fbb5daca22883478f P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:32.406229 20044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8784b9603f8544b79373158aace063af. Latest consensus state: current_term: 6 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:32.411168 20044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: LEADER
W20250111 02:57:32.414333 18615 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 7ea78feee64340e39404cba0f1bf04db (table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7ea78feee64340e39404cba0f1bf04db consensus state has no leader
W20250111 02:57:32.392627 18630 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.66:37049 (request call id 40) took 4367 ms (client timeout 1000 ms). Trace:
I20250111 02:57:32.464951 19330 tablet_bootstrap.cc:654] T e46894710a72407993b36801f3b90eb0 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:32.468017 20035 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:32.469027 20035 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:32.471637 19683 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 6 FOLLOWER]: Rejecting Update request from peer c34b72b7c70343d3b6f3fae79ce4be6c for earlier term 5. Current term is 6. Ops: []
I20250111 02:57:32.476120 19196 raft_consensus.cc:2463] T ffd063f8889b4bcdb9656466f0b144f3 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:32.477836 18986 leader_election.cc:304] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:32.478801 19923 consensus_queue.cc:1035] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:32.479744 20035 consensus_queue.cc:1046] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 }, Status: INVALID_TERM, Last received: 5.274, Next index: 275, Last known committed idx: 272, Time since last communication: 0.001s
I20250111 02:57:32.480444 19923 raft_consensus.cc:2798] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:32.480984 19923 raft_consensus.cc:695] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:32.482018 19923 consensus_queue.cc:237] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.488909 20059 raft_consensus.cc:491] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:32.488504 19321 tablet_bootstrap.cc:654] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:32.489470 20059 raft_consensus.cc:513] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.503213 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:32.498943 20059 leader_election.cc:290] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:32.506619 19133 leader_election.cc:343] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:32.510758 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:32.521281 19131 leader_election.cc:343] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:32.521708 19131 leader_election.cc:304] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:32.524569 19322 tablet_bootstrap.cc:492] T 0158a8849bf44a4fbb5daca22883478f P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:32.525097 19322 ts_tablet_manager.cc:1397] T 0158a8849bf44a4fbb5daca22883478f P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.257s user 0.010s sys 0.007s
I20250111 02:57:32.527745 18622 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet cc7158a7fc624fa4bc816ac3721c610d (table default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605]). Expected version 2 got 0
I20250111 02:57:32.528749 19330 tablet_bootstrap.cc:492] T e46894710a72407993b36801f3b90eb0 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:32.529145 19330 ts_tablet_manager.cc:1397] T e46894710a72407993b36801f3b90eb0 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.182s user 0.007s sys 0.004s
W20250111 02:57:32.420037 18630 rpcz_store.cc:269] 0111 02:57:28.025120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:28.813981 (+788861us) service_pool.cc:225] Handling call
0111 02:57:28.832662 (+ 18681us) spinlock_profiling.cc:243] Waited 7.41 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d51dee 00007fd427aaffc4 00007fd427aaff52 00007fd427aafe81 00007fd427aafdfc 00007fd427aaf0b2 00007fd425d79dfc 00007fd425d79236
0111 02:57:32.334436 (+3501774us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:32.334458 (+ 22us) write_op.cc:454] Released schema lock
0111 02:57:32.392523 (+ 58065us) inbound_call.cc:173] Queueing failure response
Related trace 'op':
0111 02:57:32.334233 (+ 0us) op_driver.cc:389] HandleFailure(Service unavailable: leader is not yet ready)
0111 02:57:32.334270 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:32.334291 (+ 21us) write_op.cc:454] Released schema lock
0111 02:57:32.334315 (+ 24us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":555,"reported_tablets":16,"spinlock_wait_cycles":17053568,"child_traces":[["op",{}]]}
I20250111 02:57:32.532312 18622 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet a983a670c2104398ac1a159ac3a1da7c (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19]). Expected version 4 got 0
I20250111 02:57:32.536461 18622 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 7ea78feee64340e39404cba0f1bf04db (table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5]). Expected version 1 got 0
I20250111 02:57:32.539481 20059 raft_consensus.cc:2743] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:32.561190 19322 raft_consensus.cc:357] T 0158a8849bf44a4fbb5daca22883478f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.561947 19322 raft_consensus.cc:383] T 0158a8849bf44a4fbb5daca22883478f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:32.562206 19322 raft_consensus.cc:738] T 0158a8849bf44a4fbb5daca22883478f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:32.583434 20058 raft_consensus.cc:491] T 855dac47872d4c5fbf8f5e38b588c7fe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:32.584043 20058 raft_consensus.cc:513] T 855dac47872d4c5fbf8f5e38b588c7fe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:32.586992 20058 leader_election.cc:290] T 855dac47872d4c5fbf8f5e38b588c7fe P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:32.588063 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855dac47872d4c5fbf8f5e38b588c7fe" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:32.588553 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855dac47872d4c5fbf8f5e38b588c7fe" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:32.590368 18841 leader_election.cc:304] T 855dac47872d4c5fbf8f5e38b588c7fe P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:32.591873 20058 raft_consensus.cc:2743] T 855dac47872d4c5fbf8f5e38b588c7fe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:32.599658 19955 tablet_loader.cc:96] loaded metadata for tablet 0158a8849bf44a4fbb5daca22883478f (table default.Table_c2fa721b96ba4e9ca293ff4b5c455447 [id=64d78652fc6f44778133bd3157964db5])
I20250111 02:57:32.600210 19997 raft_consensus.cc:3049] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 5 LEADER]: Stepping down as leader of term 5
I20250111 02:57:32.599903 19330 raft_consensus.cc:357] T e46894710a72407993b36801f3b90eb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:32.600641 19330 raft_consensus.cc:383] T e46894710a72407993b36801f3b90eb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:32.600929 19330 raft_consensus.cc:738] T e46894710a72407993b36801f3b90eb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:32.600791 19997 raft_consensus.cc:738] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c34b72b7c70343d3b6f3fae79ce4be6c, State: Running, Role: LEADER
I20250111 02:57:32.601178 19955 tablet_loader.cc:96] loaded metadata for tablet 0255034337f24337be838425136d8b4c (table default.Table_813d4e8bb0f34a46bf96ed3bee6ca86f [id=d790288a097a4416b68b93db6ac847fd])
I20250111 02:57:32.601599 19330 consensus_queue.cc:260] T e46894710a72407993b36801f3b90eb0 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:32.602094 19997 consensus_queue.cc:260] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 274, Committed index: 274, Last appended: 5.274, Last appended by leader: 274, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:32.603986 19997 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 5 FOLLOWER]: Advancing to term 6
I20250111 02:57:32.604398 19330 ts_tablet_manager.cc:1428] T e46894710a72407993b36801f3b90eb0 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.075s user 0.006s sys 0.001s
I20250111 02:57:32.613770 18622 catalog_manager.cc:5526] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa reported cstate change: term changed from 0 to 1, leader changed from <none> to d0cedab82b154d62b78809edc6f28bfa (127.18.20.67). New cstate: current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:32.562819 19322 consensus_queue.cc:260] T 0158a8849bf44a4fbb5daca22883478f P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.620744 19322 ts_tablet_manager.cc:1428] T 0158a8849bf44a4fbb5daca22883478f P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.095s user 0.004s sys 0.003s
I20250111 02:57:32.630558 19321 tablet_bootstrap.cc:492] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:32.631035 19321 ts_tablet_manager.cc:1397] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.246s user 0.010s sys 0.007s
I20250111 02:57:32.616313 18622 catalog_manager.cc:5526] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa reported cstate change: term changed from 0 to 1, leader changed from <none> to d0cedab82b154d62b78809edc6f28bfa (127.18.20.67). New cstate: current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:32.641006 18622 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 855dac47872d4c5fbf8f5e38b588c7fe (table default.Table_49e4dd58d29f4cb6a0ed4f85138713b8 [id=f1373922978a4598b888e1863525942e]). Expected version 1 got 0
I20250111 02:57:32.644201 18622 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 845da3ee0863430ea3fe9de8c84f2c17 (table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3]). Expected version 2 got 0
I20250111 02:57:32.647455 18622 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]). Expected version 4 got 0
I20250111 02:57:32.653864 19330 tablet_bootstrap.cc:492] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:32.657274 18900 raft_consensus.cc:3054] T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:32.663503 19322 tablet_bootstrap.cc:492] T 62c6b166a29943ceb15f093fac728c79 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:32.682718 19955 tablet_loader.cc:96] loaded metadata for tablet 07f15096ea484b499fbdecec74eb6f66 (table default.Table_7653ac6289d7467590ac03b43df1b8bf [id=043fd6ad4f5c4a2685af5bbecda90328])
I20250111 02:57:32.684054 19955 tablet_loader.cc:96] loaded metadata for tablet 0b6687f39d4246b49c0c3eb904b11c1a (table default.Table_857f04d0de7e47a4828db9a8997f1009 [id=9cf908e0c7d44b60b88a32cf2d6368a0])
I20250111 02:57:32.634375 19321 raft_consensus.cc:357] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.695902 19321 raft_consensus.cc:383] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:32.696251 19321 raft_consensus.cc:738] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:32.696985 19321 consensus_queue.cc:260] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.702697 19330 tablet_bootstrap.cc:654] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:32.723456 19321 ts_tablet_manager.cc:1428] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.092s user 0.006s sys 0.001s
I20250111 02:57:32.724541 19321 tablet_bootstrap.cc:492] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:32.735977 19955 tablet_loader.cc:96] loaded metadata for tablet 0c669a3ff8aa46fa8b04b86ed3bfde49 (table default.Table_fc5930652905494584fde76fff8629d8 [id=a25d2996d4d64d91980bb5a067f5b118])
I20250111 02:57:32.737902 19955 tablet_loader.cc:96] loaded metadata for tablet 0d4ba33836f04ed19d67b8bc161e57c4 (table default.Table_bf5cf724093d4691a60379b87cfab2f0 [id=c151a212f267426a9cb6180c938a5815])
I20250111 02:57:32.751410 19322 tablet_bootstrap.cc:654] T 62c6b166a29943ceb15f093fac728c79 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:32.771749 20058 raft_consensus.cc:491] T 0158a8849bf44a4fbb5daca22883478f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:32.772348 20058 raft_consensus.cc:513] T 0158a8849bf44a4fbb5daca22883478f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.776479 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0158a8849bf44a4fbb5daca22883478f" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:32.778559 19955 tablet_loader.cc:96] loaded metadata for tablet 12065effb54b4d4a94349641109b578c (table default.Table_d7e00ace61e545129a2e99093dba16db [id=d0e7184505c24fc5b9960e8b6d1c955d])
I20250111 02:57:32.778716 19754 raft_consensus.cc:491] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:32.779320 19754 raft_consensus.cc:513] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:32.784617 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0158a8849bf44a4fbb5daca22883478f" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:32.787708 18842 leader_election.cc:304] T 0158a8849bf44a4fbb5daca22883478f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:32.786747 19754 leader_election.cc:290] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:32.790061 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46894710a72407993b36801f3b90eb0" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:32.790989 19322 tablet_bootstrap.cc:492] T 62c6b166a29943ceb15f093fac728c79 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:32.791486 19322 ts_tablet_manager.cc:1397] T 62c6b166a29943ceb15f093fac728c79 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.128s user 0.012s sys 0.000s
I20250111 02:57:32.791913 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46894710a72407993b36801f3b90eb0" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:32.793747 19133 leader_election.cc:343] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:32.800974 19321 tablet_bootstrap.cc:654] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:32.790868 19049 raft_consensus.cc:2463] T e46894710a72407993b36801f3b90eb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:32.804154 19131 leader_election.cc:304] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:32.805284 20059 raft_consensus.cc:491] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:32.805785 20059 raft_consensus.cc:513] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.808211 20059 leader_election.cc:290] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:32.808979 20059 raft_consensus.cc:2798] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:32.809376 20059 raft_consensus.cc:491] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:32.810120 20059 raft_consensus.cc:3054] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:32.774839 20058 leader_election.cc:290] T 0158a8849bf44a4fbb5daca22883478f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:32.813295 20058 raft_consensus.cc:2743] T 0158a8849bf44a4fbb5daca22883478f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:32.813472 19955 tablet_loader.cc:96] loaded metadata for tablet 16d88c5da4f345c3a43a1ef00f82856b (table default.Table_a7b2b5af5dfb439a84159b75de0d03b1 [id=dc7854580f744d4083f6aa3f0f0f7bf2])
I20250111 02:57:32.817133 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "061e6eb5775443679e9cbd6b11f66fe3" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:32.818929 19133 leader_election.cc:343] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:32.819943 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "061e6eb5775443679e9cbd6b11f66fe3" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:32.823005 19595 leader_election.cc:336] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593): Timed out: RequestConsensusVote RPC to 127.18.20.124:34593 timed out after 0.520s (SENT)
I20250111 02:57:32.823601 19595 leader_election.cc:304] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727; no voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c
W20250111 02:57:32.834344 19131 leader_election.cc:343] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:32.834772 19131 leader_election.cc:304] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:32.794564 19322 raft_consensus.cc:357] T 62c6b166a29943ceb15f093fac728c79 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.846280 19322 raft_consensus.cc:383] T 62c6b166a29943ceb15f093fac728c79 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:32.839025 19754 raft_consensus.cc:2743] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:32.846671 19322 raft_consensus.cc:738] T 62c6b166a29943ceb15f093fac728c79 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:32.847352 19322 consensus_queue.cc:260] T 62c6b166a29943ceb15f093fac728c79 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.849977 19322 ts_tablet_manager.cc:1428] T 62c6b166a29943ceb15f093fac728c79 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.058s user 0.005s sys 0.000s
I20250111 02:57:32.863198 19955 tablet_loader.cc:96] loaded metadata for tablet 1b69a8c2caea4bddac3f539733bcd3af (table default.Table_bf50e4c0392544f8a24b861d3edaf46e [id=4b08fa373b0b460094c09b4d3353951c])
I20250111 02:57:32.865186 19955 tablet_loader.cc:96] loaded metadata for tablet 1ef92ac716824107916a1ea4fd49d5aa (table default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609])
I20250111 02:57:32.868362 19321 tablet_bootstrap.cc:492] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:32.868855 19321 ts_tablet_manager.cc:1397] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.144s user 0.007s sys 0.004s
I20250111 02:57:32.850973 19322 tablet_bootstrap.cc:492] T 220da826a0f140d59193e8ad72096b60 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:32.886513 19321 raft_consensus.cc:357] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.887296 19321 raft_consensus.cc:383] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:32.887554 19321 raft_consensus.cc:738] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:32.888170 19321 consensus_queue.cc:260] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.890357 19321 ts_tablet_manager.cc:1428] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.021s user 0.004s sys 0.001s
I20250111 02:57:32.902638 19330 tablet_bootstrap.cc:492] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:32.903115 19330 ts_tablet_manager.cc:1397] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.249s user 0.016s sys 0.009s
I20250111 02:57:32.905835 19330 raft_consensus.cc:357] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.908761 19330 raft_consensus.cc:383] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:32.909065 19330 raft_consensus.cc:738] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:32.909775 19330 consensus_queue.cc:260] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.911535 19321 tablet_bootstrap.cc:492] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:32.932621 19330 ts_tablet_manager.cc:1428] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.029s user 0.004s sys 0.001s
I20250111 02:57:32.933624 19330 tablet_bootstrap.cc:492] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:32.936303 20059 raft_consensus.cc:513] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:32.956990 19330 tablet_bootstrap.cc:654] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:32.960295 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46894710a72407993b36801f3b90eb0" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:32.960942 19049 raft_consensus.cc:3054] T e46894710a72407993b36801f3b90eb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:32.880919 19955 tablet_loader.cc:96] loaded metadata for tablet 2009d6f54e2842e8be3af2cc9147dc2c (table default.Table_24e5a7f819cc45dbb7efbd7286ad7005 [id=c04c27fb52054de185e1f3075f89a59a])
I20250111 02:57:32.971343 19955 tablet_loader.cc:96] loaded metadata for tablet 218b87091d9546ba859f954a8f9e089e (table default.Table_c398ec562f594aca8c1e62cde38c04f9 [id=14bc02cbe30c486782d8fa4396855a9a])
I20250111 02:57:32.972750 19955 tablet_loader.cc:96] loaded metadata for tablet 220da826a0f140d59193e8ad72096b60 (table default.Table_5e7c139097914dd7b7c8e55058d5290d [id=9143d1d3b8754ac78d615a1cc4601d86])
I20250111 02:57:32.974014 19955 tablet_loader.cc:96] loaded metadata for tablet 237ea33846044127987c1d81a3203553 (table default.Table_b9769d513090402bb596c949defad0df [id=76e72ea89f6e408fbaeda0efc7e79770])
I20250111 02:57:32.980434 19049 raft_consensus.cc:2463] T e46894710a72407993b36801f3b90eb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:32.981798 19131 leader_election.cc:304] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:32.982901 19754 raft_consensus.cc:2798] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:32.983155 20055 raft_consensus.cc:491] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:32.983604 20055 raft_consensus.cc:513] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:32.987799 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea76b354a75041e888cf862250e4947b" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:32.988190 20055 leader_election.cc:290] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:32.988480 19196 raft_consensus.cc:2463] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:32.989905 18986 leader_election.cc:304] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:32.990845 20055 raft_consensus.cc:2798] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:32.991346 20055 raft_consensus.cc:491] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:32.991765 20055 raft_consensus.cc:3054] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:32.995306 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46894710a72407993b36801f3b90eb0" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:33.000360 19754 raft_consensus.cc:695] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:33.001595 19754 consensus_queue.cc:237] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:33.022086 19133 leader_election.cc:343] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:33.023393 20059 leader_election.cc:290] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:33.027833 19330 tablet_bootstrap.cc:492] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:33.028354 19330 ts_tablet_manager.cc:1397] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.095s user 0.012s sys 0.001s
I20250111 02:57:33.029420 19997 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:33.030059 19997 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:33.033025 19997 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:33.034255 18639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 274 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:33.030934 19330 raft_consensus.cc:357] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.039845 19330 raft_consensus.cc:383] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:33.040161 19330 raft_consensus.cc:738] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:33.041424 20090 raft_consensus.cc:491] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:33.041934 20090 raft_consensus.cc:513] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:33.044314 20083 log.cc:926] Time spent T 5cf1ce65ccf54358b95665b9764fbdb0 P 8221604733fe4fc0a1fc32de8d6af71c: Append to log took a long time: real 0.141s user 0.000s sys 0.000s
I20250111 02:57:33.046537 19322 tablet_bootstrap.cc:654] T 220da826a0f140d59193e8ad72096b60 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:33.050812 19683 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 6 FOLLOWER]: Refusing update from remote peer 8784b9603f8544b79373158aace063af: Log matching property violated. Preceding OpId in replica: term: 5 index: 274. Preceding OpId from leader: term: 6 index: 276. (index mismatch)
I20250111 02:57:33.051766 19682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 274 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:33.052383 19682 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c34b72b7c70343d3b6f3fae79ce4be6c in term 6.
I20250111 02:57:33.061022 20044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 272, Time since last communication: 0.000s
I20250111 02:57:33.061592 19955 tablet_loader.cc:96] loaded metadata for tablet 29f27ed2c53d43d7a51ee2e45afaf9fc (table default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7])
I20250111 02:57:33.063164 18695 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 8784b9603f8544b79373158aace063af
I20250111 02:57:33.065251 19955 tablet_loader.cc:96] loaded metadata for tablet 2f11541b436e42b7b0728da2aaec48f7 (table default.Table_f36240205653470e9a67400b787ce860 [id=69f6e7b3ec40423a9d0bef5487ffe84a])
I20250111 02:57:33.048064 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26e5db463e94d919640a07a2f8574a8" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:33.069727 19997 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250111 02:57:33.070425 19133 leader_election.cc:343] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:33.070190 19997 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:33.070811 19997 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 6 FOLLOWER]: Advancing to term 7
I20250111 02:57:33.078786 20090 leader_election.cc:290] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:33.079641 20090 raft_consensus.cc:491] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:33.080835 20090 raft_consensus.cc:513] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.040786 19330 consensus_queue.cc:260] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.099709 19957 raft_consensus.cc:491] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:33.100335 19957 raft_consensus.cc:513] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.102803 19957 leader_election.cc:290] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:33.116274 19330 ts_tablet_manager.cc:1428] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.088s user 0.005s sys 0.001s
I20250111 02:57:33.117416 19330 tablet_bootstrap.cc:492] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:33.115484 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220da826a0f140d59193e8ad72096b60" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:33.117941 19192 raft_consensus.cc:2463] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:33.119769 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220da826a0f140d59193e8ad72096b60" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:33.120047 19321 tablet_bootstrap.cc:654] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:33.121096 19133 leader_election.cc:343] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:33.119532 18986 leader_election.cc:304] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:33.124039 19957 raft_consensus.cc:2798] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:33.124487 19957 raft_consensus.cc:491] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:33.124783 19957 raft_consensus.cc:3054] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:33.127370 19322 tablet_bootstrap.cc:492] T 220da826a0f140d59193e8ad72096b60 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:33.127857 19322 ts_tablet_manager.cc:1397] T 220da826a0f140d59193e8ad72096b60 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.277s user 0.018s sys 0.001s
I20250111 02:57:33.127652 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26e5db463e94d919640a07a2f8574a8" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:33.129128 19131 leader_election.cc:343] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:33.129662 19131 leader_election.cc:304] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:33.130832 19322 raft_consensus.cc:357] T 220da826a0f140d59193e8ad72096b60 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.131646 19322 raft_consensus.cc:383] T 220da826a0f140d59193e8ad72096b60 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:33.131933 19322 raft_consensus.cc:738] T 220da826a0f140d59193e8ad72096b60 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:33.132426 19957 raft_consensus.cc:513] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.132622 19322 consensus_queue.cc:260] T 220da826a0f140d59193e8ad72096b60 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.135217 19322 ts_tablet_manager.cc:1428] T 220da826a0f140d59193e8ad72096b60 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250111 02:57:33.136222 19322 tablet_bootstrap.cc:492] T e46894710a72407993b36801f3b90eb0 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:33.154541 19957 leader_election.cc:290] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:33.160918 19330 tablet_bootstrap.cc:654] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:33.162380 19322 tablet_bootstrap.cc:654] T e46894710a72407993b36801f3b90eb0 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:33.186741 19322 tablet_bootstrap.cc:492] T e46894710a72407993b36801f3b90eb0 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:33.187223 19322 ts_tablet_manager.cc:1397] T e46894710a72407993b36801f3b90eb0 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.001s
W20250111 02:57:33.192577 18842 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 404963us
I20250111 02:57:33.197288 19330 tablet_bootstrap.cc:492] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:33.197750 19330 ts_tablet_manager.cc:1397] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.081s user 0.010s sys 0.000s
I20250111 02:57:33.200426 19330 raft_consensus.cc:357] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.201174 19330 raft_consensus.cc:383] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:33.201439 19330 raft_consensus.cc:738] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
W20250111 02:57:33.193898 19595 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.124:34593, user_credentials={real_user=slave}} blocked reactor thread for 370978us
I20250111 02:57:33.189666 19322 raft_consensus.cc:357] T e46894710a72407993b36801f3b90eb0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:33.202085 19330 consensus_queue.cc:260] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.202643 19322 raft_consensus.cc:383] T e46894710a72407993b36801f3b90eb0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:33.202979 19322 raft_consensus.cc:738] T e46894710a72407993b36801f3b90eb0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:33.169512 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220da826a0f140d59193e8ad72096b60" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:33.204443 19192 raft_consensus.cc:3054] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:33.205550 19322 consensus_queue.cc:260] T e46894710a72407993b36801f3b90eb0 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:33.206070 19955 tablet_loader.cc:96] loaded metadata for tablet 315ac86194b04802909a573876fce64b (table default.Table_ea068446b5434b3ca5f0afa8386c0655 [id=dc09e842add34ef099edfd1296255c4b])
I20250111 02:57:33.213307 19322 ts_tablet_manager.cc:1428] T e46894710a72407993b36801f3b90eb0 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20250111 02:57:33.214350 19322 tablet_bootstrap.cc:492] T 237ea33846044127987c1d81a3203553 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:33.207703 20090 leader_election.cc:290] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:33.221843 20090 raft_consensus.cc:2743] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:33.222545 19955 tablet_loader.cc:96] loaded metadata for tablet 350f7c7f14004bb4bc3cb2338c7f172f (table default.Table_28d700d6f8444cfeab3dc52dd2bec17f [id=0e5eefce9dfa453eb277f75189def89e])
I20250111 02:57:33.223793 19955 tablet_loader.cc:96] loaded metadata for tablet 3514d795a0f24adcaa2accc0cf05b142 (table default.Table_4c9aebb06d554107a5448d968fdb0436 [id=3fab00618e6548de85b5d20adee4e9ac])
I20250111 02:57:33.224844 19330 ts_tablet_manager.cc:1428] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20250111 02:57:33.224956 19955 tablet_loader.cc:96] loaded metadata for tablet 36f71f8dd2894d40b3bfdd20c1453787 (table default.Table_cce40438ec8e4527b01088af6b7b0981 [id=cb42d3930a9a44bfb5dc40feb979d505])
I20250111 02:57:33.238350 19192 raft_consensus.cc:2463] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:33.239718 18986 leader_election.cc:304] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:33.240556 19923 raft_consensus.cc:2798] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:33.240799 19322 tablet_bootstrap.cc:654] T 237ea33846044127987c1d81a3203553 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:33.241016 19923 raft_consensus.cc:695] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:33.241819 19923 consensus_queue.cc:237] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.266778 19321 tablet_bootstrap.cc:492] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:33.267374 19321 ts_tablet_manager.cc:1397] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.356s user 0.003s sys 0.009s
I20250111 02:57:33.273339 19322 tablet_bootstrap.cc:492] T 237ea33846044127987c1d81a3203553 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:33.273983 19322 ts_tablet_manager.cc:1397] T 237ea33846044127987c1d81a3203553 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.060s user 0.011s sys 0.000s
I20250111 02:57:33.277392 19322 raft_consensus.cc:357] T 237ea33846044127987c1d81a3203553 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.278172 19322 raft_consensus.cc:383] T 237ea33846044127987c1d81a3203553 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:33.278441 19322 raft_consensus.cc:738] T 237ea33846044127987c1d81a3203553 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:33.279313 19322 consensus_queue.cc:260] T 237ea33846044127987c1d81a3203553 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.286518 19322 ts_tablet_manager.cc:1428] T 237ea33846044127987c1d81a3203553 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.012s user 0.002s sys 0.003s
I20250111 02:57:33.287603 19322 tablet_bootstrap.cc:492] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:33.289774 19321 raft_consensus.cc:357] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.290517 19321 raft_consensus.cc:383] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:33.299886 19322 tablet_bootstrap.cc:654] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:33.308288 20099 raft_consensus.cc:491] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:33.308904 20099 raft_consensus.cc:513] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.311829 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb96635f30b34de48870daf5166813bc" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:33.312695 19322 tablet_bootstrap.cc:492] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:33.313148 19322 ts_tablet_manager.cc:1397] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.003s
W20250111 02:57:33.313306 19133 leader_election.cc:343] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:33.314376 20099 leader_election.cc:290] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:33.027312 20055 raft_consensus.cc:513] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.315388 19322 raft_consensus.cc:357] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:33.316149 19330 tablet_bootstrap.cc:492] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:33.316134 19322 raft_consensus.cc:383] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:33.316380 19322 raft_consensus.cc:738] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:33.316958 19322 consensus_queue.cc:260] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:33.322507 20086 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250111 02:57:33.323968 19322 ts_tablet_manager.cc:1428] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250111 02:57:33.324978 19322 tablet_bootstrap.cc:492] T 3052e9707de14e1f9e38236d56908ed3 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:33.334791 20095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8784b9603f8544b79373158aace063af. Latest consensus state: current_term: 6 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:33.335857 20095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:33.337178 20096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:33.339198 20096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:33.347558 19955 tablet_loader.cc:96] loaded metadata for tablet 3c57d85390324bbc9ea30a709faec01c (table default.Table_52d947397a774f96a4fdb5087da4e19a [id=7196ac9c0c4d4dada6c94803e1220caa])
I20250111 02:57:33.348788 19955 tablet_loader.cc:96] loaded metadata for tablet 3d873a0faf4a4645a3969067ae0583dc (table default.Table_c9582d98645645f3aa80a65f7652cba8 [id=2d5d8f8a7ade443f91a7da34add25a91])
I20250111 02:57:33.356750 19923 raft_consensus.cc:491] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:33.357611 19923 raft_consensus.cc:513] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.357475 19322 tablet_bootstrap.cc:654] T 3052e9707de14e1f9e38236d56908ed3 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:33.362169 19923 leader_election.cc:290] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:33.343258 20103 log.cc:926] Time spent T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20250111 02:57:33.377879 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea76b354a75041e888cf862250e4947b" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:33.379245 19863 log.cc:926] Time spent T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Append to log took a long time: real 0.103s user 0.000s sys 0.002s
I20250111 02:57:33.380996 20055 leader_election.cc:290] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:33.381713 19955 tablet_loader.cc:96] loaded metadata for tablet 40fdec982cf643c68ef90798f4f65ad4 (table default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927])
I20250111 02:57:33.383370 19330 tablet_bootstrap.cc:654] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:33.384202 18900 raft_consensus.cc:3054] T 855dac47872d4c5fbf8f5e38b588c7fe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:33.387588 19955 tablet_loader.cc:96] loaded metadata for tablet 49616e4e7cc647509e72cde57b87d6fa (table default.Table_c4faffc9659848ea9bff581e8d88d40e [id=e709d8f708144de8a18d241f0e0306a0])
I20250111 02:57:33.394920 19997 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:33.372967 20100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:33.397707 20100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:33.405938 20055 consensus_queue.cc:1035] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:33.413812 19923 consensus_queue.cc:1035] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:33.290817 19321 raft_consensus.cc:738] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:33.342411 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea76b354a75041e888cf862250e4947b" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:33.427318 19321 consensus_queue.cc:260] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.427909 19196 raft_consensus.cc:3054] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:33.396708 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 267628us
I20250111 02:57:33.429665 19321 ts_tablet_manager.cc:1428] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.162s user 0.004s sys 0.001s
I20250111 02:57:33.431533 19321 tablet_bootstrap.cc:492] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
W20250111 02:57:33.432586 18615 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet dea4db154779483cb069118e3f3789b0 (table default.Table_8eae98e1e49c4f6c9e52644256812eaa [id=7dfae414add74b6491f2f7595a551342], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet dea4db154779483cb069118e3f3789b0 has no consensus state
I20250111 02:57:33.361619 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3052e9707de14e1f9e38236d56908ed3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:33.433594 19192 raft_consensus.cc:2463] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:33.417119 19330 tablet_bootstrap.cc:492] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:33.435052 18986 leader_election.cc:304] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:33.444015 19330 ts_tablet_manager.cc:1397] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.128s user 0.008s sys 0.006s
I20250111 02:57:33.444404 20054 raft_consensus.cc:2798] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:33.444959 20054 raft_consensus.cc:491] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:33.445046 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220da826a0f140d59193e8ad72096b60" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:33.445492 20054 raft_consensus.cc:3054] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:33.446646 19330 raft_consensus.cc:357] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.447358 19330 raft_consensus.cc:383] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:33.447618 19330 raft_consensus.cc:738] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:33.448858 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb96635f30b34de48870daf5166813bc" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:33.448267 19330 consensus_queue.cc:260] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.452343 19131 leader_election.cc:304] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:33.454492 19330 ts_tablet_manager.cc:1428] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250111 02:57:33.455247 20054 raft_consensus.cc:513] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.456148 19330 tablet_bootstrap.cc:492] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:33.456341 18906 raft_consensus.cc:3054] T 62c6b166a29943ceb15f093fac728c79 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:33.457765 20054 leader_election.cc:290] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:33.466419 18615 leader_election.cc:336] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593): Timed out: RequestConsensusVote RPC to 127.18.20.124:34593 timed out after 0.665s (SENT)
W20250111 02:57:33.469931 19131 leader_election.cc:343] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:33.470765 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220da826a0f140d59193e8ad72096b60" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:33.400563 19955 tablet_loader.cc:96] loaded metadata for tablet 56fb0bf949904208af003f08caae7594 (table default.Table_23a3f1c40c15458e93c2ef79a3c9e9e2 [id=c12add3740174f66996466994ebe9971])
I20250111 02:57:33.472954 19955 tablet_loader.cc:96] loaded metadata for tablet 5a4bc8e9c0c947f6861a01d3e3bd7f31 (table default.Table_a68194fdb909449f878695d09b81c753 [id=49aa9df32f8e4001b66d751832a64534])
I20250111 02:57:33.476594 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 274 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727"
I20250111 02:57:33.487933 19330 tablet_bootstrap.cc:654] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:33.490386 19955 tablet_loader.cc:96] loaded metadata for tablet 63aef98c83d94918bb3163d491870084 (table default.Table_98958a28faeb43e8a3b34fcf20d936aa [id=98e24956666a413cbf4564438a594ef0])
I20250111 02:57:33.491669 19955 tablet_loader.cc:96] loaded metadata for tablet 64abb30da7fc452cad774bbe59b498f4 (table default.Table_26836b21f0f84973b442a4966f97d1ee [id=0037843fe90a422399b837b548ff45b9])
I20250111 02:57:33.496207 19131 leader_election.cc:304] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:33.481047 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220da826a0f140d59193e8ad72096b60" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:33.498354 18901 raft_consensus.cc:3054] T 220da826a0f140d59193e8ad72096b60 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:33.492787 19955 tablet_loader.cc:96] loaded metadata for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038])
I20250111 02:57:33.502132 19330 tablet_bootstrap.cc:492] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:33.502575 19330 ts_tablet_manager.cc:1397] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.002s
I20250111 02:57:33.505592 19330 raft_consensus.cc:357] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.506306 19330 raft_consensus.cc:383] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:33.506596 19330 raft_consensus.cc:738] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:33.508127 18901 raft_consensus.cc:2463] T 220da826a0f140d59193e8ad72096b60 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:33.511308 19955 tablet_loader.cc:96] loaded metadata for tablet 6ff06134303d4cb0b4292fe1c0c2d443 (table default.Table_2c181f66f28741b7b388a7b563b7d2f5 [id=ac0e88a97cf740fd8df3e6cbd81888b5])
I20250111 02:57:33.515944 19322 tablet_bootstrap.cc:492] T 3052e9707de14e1f9e38236d56908ed3 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:33.516436 19322 ts_tablet_manager.cc:1397] T 3052e9707de14e1f9e38236d56908ed3 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.192s user 0.030s sys 0.018s
I20250111 02:57:33.516103 19330 consensus_queue.cc:260] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.518456 19330 ts_tablet_manager.cc:1428] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250111 02:57:33.522677 19322 raft_consensus.cc:357] T 3052e9707de14e1f9e38236d56908ed3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.523504 19322 raft_consensus.cc:383] T 3052e9707de14e1f9e38236d56908ed3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:33.523785 19322 raft_consensus.cc:738] T 3052e9707de14e1f9e38236d56908ed3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:33.524482 19322 consensus_queue.cc:260] T 3052e9707de14e1f9e38236d56908ed3 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.527235 19322 ts_tablet_manager.cc:1428] T 3052e9707de14e1f9e38236d56908ed3 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250111 02:57:33.528262 19322 tablet_bootstrap.cc:492] T ffd063f8889b4bcdb9656466f0b144f3 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:33.531217 19955 tablet_loader.cc:96] loaded metadata for tablet 7145efe5c0094b70be2cd6481b770248 (table default.Table_7ee7d20cfb7e4985bdd8539d15e354fe [id=ff66cd83751a45a09d8012cc8a9f1758])
I20250111 02:57:33.537943 19330 tablet_bootstrap.cc:492] T e4bd42ad66064bcaa043746081f1c759 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:33.546774 19955 tablet_loader.cc:96] loaded metadata for tablet 788c34eff1884d979add7e09615c02ae (table default.Table_dfe42fac0a764052a8d2f6add8882b64 [id=b42908f2a2c14d0a84027cb7a1d775b7])
I20250111 02:57:33.518852 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3052e9707de14e1f9e38236d56908ed3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:33.547792 19194 raft_consensus.cc:3054] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:33.549074 19955 tablet_loader.cc:96] loaded metadata for tablet 79f4500d3beb4c9bb20f84c661c3758b (table default.Table_bdacd508498241408a549fdbc6704512 [id=46a029216fdb4496bf9df1d776ccdf7e])
W20250111 02:57:33.553632 18615 leader_election.cc:336] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593): Timed out: RequestConsensusVote RPC to 127.18.20.124:34593 timed out after 0.614s (SENT)
I20250111 02:57:33.555449 19194 raft_consensus.cc:2463] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:33.520977 19321 tablet_bootstrap.cc:654] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:33.489116 19754 raft_consensus.cc:2743] T 220da826a0f140d59193e8ad72096b60 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:33.560206 18639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 274 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af"
I20250111 02:57:33.561957 18693 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:33.572439 19955 tablet_loader.cc:96] loaded metadata for tablet 7c3c5b038d994d07abfea701194e8ad9 (table default.Table_bdf982e1d6904892ac08ff16ed20a636 [id=a2b6467b58754b85ab0481a9e8aa0ace])
I20250111 02:57:33.512820 18906 raft_consensus.cc:2371] T 220da826a0f140d59193e8ad72096b60 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1. Re-sending same reply.
I20250111 02:57:33.573793 19955 tablet_loader.cc:96] loaded metadata for tablet 7ea78feee64340e39404cba0f1bf04db (table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5])
I20250111 02:57:33.544059 19330 tablet_bootstrap.cc:654] T e4bd42ad66064bcaa043746081f1c759 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:33.574971 19955 tablet_loader.cc:96] loaded metadata for tablet 8414653def014ce98bc38d87aac75bae (table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204])
I20250111 02:57:33.575551 18903 raft_consensus.cc:3054] T 0158a8849bf44a4fbb5daca22883478f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:33.593519 20084 raft_consensus.cc:491] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:33.594022 20084 raft_consensus.cc:513] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.594504 19322 tablet_bootstrap.cc:654] T ffd063f8889b4bcdb9656466f0b144f3 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:33.597685 19955 tablet_loader.cc:96] loaded metadata for tablet 845da3ee0863430ea3fe9de8c84f2c17 (table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3])
I20250111 02:57:33.598747 20084 leader_election.cc:290] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:33.599159 19955 tablet_loader.cc:96] loaded metadata for tablet 855dac47872d4c5fbf8f5e38b588c7fe (table default.Table_49e4dd58d29f4cb6a0ed4f85138713b8 [id=f1373922978a4598b888e1863525942e])
I20250111 02:57:33.599155 19804 mvcc.cc:204] Tried to move back new op lower bound from 7112967147242586112 to 7112967095606104064. Current Snapshot: MvccSnapshot[applied={T|T < 7112967146203332608 or (T in {7112967146489458688,7112967146559328256,7112967147189145600})}]
I20250111 02:57:33.602008 19955 tablet_loader.cc:96] loaded metadata for tablet 8d20e8099b964fbc9e8e41447772d8d5 (table default.Table_a02982c32dcb409385f8f4e99b95423f [id=354a3a4b0b774fb9bc83623b2cb87f8a])
I20250111 02:57:33.604802 19330 tablet_bootstrap.cc:492] T e4bd42ad66064bcaa043746081f1c759 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:33.605230 19330 ts_tablet_manager.cc:1397] T e4bd42ad66064bcaa043746081f1c759 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.004s
W20250111 02:57:33.606388 18841 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Not found: Tablet not found: 32afc2f158254811836fcf00f7f409ed
I20250111 02:57:33.611331 19754 raft_consensus.cc:2743] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:33.613459 19754 raft_consensus.cc:491] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:33.614935 20084 raft_consensus.cc:491] T e46894710a72407993b36801f3b90eb0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:33.615403 20084 raft_consensus.cc:513] T e46894710a72407993b36801f3b90eb0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:33.619273 20084 leader_election.cc:290] T e46894710a72407993b36801f3b90eb0 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:33.614288 19754 raft_consensus.cc:513] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.620208 19955 tablet_loader.cc:96] loaded metadata for tablet 8eb87813c28742edad99470c84269ef4 (table default.Table_f9181ca352384593b93119fb3e0cd812 [id=e0ac9d601c464a3daaaa63c4cb90e471])
I20250111 02:57:33.622197 19955 tablet_loader.cc:96] loaded metadata for tablet 900d4d665c4a4048bc6989ba63391cc1 (table default.Table_baa2b0022e914aa5b485db90037be4af [id=3f31cb2959f340e3ae2656b63eaa52e6])
I20250111 02:57:33.623605 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf61f6cb63d4e8ea46b440e7ecce00d" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:33.624415 19955 tablet_loader.cc:96] loaded metadata for tablet 9192160e931445f1acd207604192f759 (table default.Table_bccba04b48cc408d845624ee7ed98e5f [id=1496848ec9594cad889312978436d9c4])
W20250111 02:57:33.625136 19133 leader_election.cc:343] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:33.626246 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46894710a72407993b36801f3b90eb0" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:33.626564 19955 tablet_loader.cc:96] loaded metadata for tablet 933828fd6fb041e9a4889f4ea8a3c5ff (table default.Table_dc22147bd8ba4ea18bc4f41b25b277c0 [id=1d14d647d8c24e31a84ce6ac7e2c217e])
I20250111 02:57:33.622839 19754 leader_election.cc:290] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:33.635564 20112 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250111 02:57:33.638815 19321 tablet_bootstrap.cc:492] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:33.639366 19321 ts_tablet_manager.cc:1397] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.208s user 0.005s sys 0.006s
I20250111 02:57:33.641932 19321 raft_consensus.cc:357] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.642722 19321 raft_consensus.cc:383] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:33.657019 19330 raft_consensus.cc:357] T e4bd42ad66064bcaa043746081f1c759 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:33.657986 19330 raft_consensus.cc:383] T e4bd42ad66064bcaa043746081f1c759 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:33.658339 19330 raft_consensus.cc:738] T e4bd42ad66064bcaa043746081f1c759 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:33.659440 19330 consensus_queue.cc:260] T e4bd42ad66064bcaa043746081f1c759 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:33.662233 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3052e9707de14e1f9e38236d56908ed3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:33.662482 19330 ts_tablet_manager.cc:1428] T e4bd42ad66064bcaa043746081f1c759 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.057s user 0.007s sys 0.001s
I20250111 02:57:33.662889 18908 raft_consensus.cc:2463] T 3052e9707de14e1f9e38236d56908ed3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:33.666795 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea76b354a75041e888cf862250e4947b" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:33.658170 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 113834us
I20250111 02:57:33.668777 18986 leader_election.cc:304] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:33.670377 20046 raft_consensus.cc:2798] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:33.670840 20046 raft_consensus.cc:695] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:33.672200 20046 consensus_queue.cc:237] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:33.629343 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 159468us
I20250111 02:57:33.642992 19321 raft_consensus.cc:738] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:33.691972 19321 consensus_queue.cc:260] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.694504 19321 ts_tablet_manager.cc:1428] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.055s user 0.004s sys 0.000s
W20250111 02:57:33.640066 18693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 78330.4us
W20250111 02:57:33.694643 19863 log.cc:926] Time spent T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Append to log took a long time: real 0.268s user 0.002s sys 0.001s
I20250111 02:57:33.701645 20055 raft_consensus.cc:491] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:33.702486 20055 raft_consensus.cc:513] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.643424 19955 tablet_loader.cc:96] loaded metadata for tablet 934c900baf1b4000af7d7ec60cbb3591 (table default.Table_4f96c1244592444a994b5c5631eaaadc [id=4c817a432cf649799a2589f12a320ebd])
I20250111 02:57:33.658890 19997 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 7 election: Requested vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:33.709563 19955 tablet_loader.cc:96] loaded metadata for tablet 94bb9089c71b49bea9c0f4d94fc22fb1 (table default.Table_b19a186d285d4b5b845727344f7d155b [id=b28b429fe5844cd6ac6f4de349c9449b])
I20250111 02:57:33.716842 19955 tablet_loader.cc:96] loaded metadata for tablet 9d6de202f4e547a0b8b90cfa09ad319a (table default.Table_6823aef149894f49822fe303c036446d [id=d5ffe760744748a98f80f25b8092f9fa])
I20250111 02:57:33.718055 19955 tablet_loader.cc:96] loaded metadata for tablet 9ee884f4d31f46c197c99bf72f1f19bf (table default.Table_85a0376bcaf24d0081e780e8c19764b8 [id=5f97e9e0988848319a9c8aaca6aad982])
W20250111 02:57:33.718565 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 49856.7us
I20250111 02:57:33.721374 19955 tablet_loader.cc:96] loaded metadata for tablet a2eae136623e4802b083d6526c6867c9 (table default.Table_97e52a18fbd44f189d46ba9b26cd9033 [id=566e89e7552f4ce08fba16fe299b9ccf])
I20250111 02:57:33.722681 19955 tablet_loader.cc:96] loaded metadata for tablet a6e4b222bf974a8d80b25141a5686761 (table default.Table_5e21879e67544c71af627f795fd6d940 [id=d6c0153a97cd42d49f534f48fc5b8287])
I20250111 02:57:33.723887 19955 tablet_loader.cc:96] loaded metadata for tablet a983a670c2104398ac1a159ac3a1da7c (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19])
I20250111 02:57:33.725220 19955 tablet_loader.cc:96] loaded metadata for tablet aaab428f798a44f69c80e5792d56b920 (table default.Table_e42719b273bf4b8f96fa48d1a3d6c8d8 [id=27f7d429d88b45b8b76d416b9dc582db])
I20250111 02:57:33.726497 19955 tablet_loader.cc:96] loaded metadata for tablet aca35147034446b3882b5244923bddc7 (table default.Table_1431a031eea3496b9764a373a012a367 [id=a0cdf8fe647d4c84a9ce97df9ce9a0ce])
I20250111 02:57:33.732452 19322 tablet_bootstrap.cc:492] T ffd063f8889b4bcdb9656466f0b144f3 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:33.732976 19322 ts_tablet_manager.cc:1397] T ffd063f8889b4bcdb9656466f0b144f3 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.205s user 0.013s sys 0.016s
I20250111 02:57:33.709589 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3052e9707de14e1f9e38236d56908ed3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:33.739451 19321 tablet_bootstrap.cc:492] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:33.741310 20055 leader_election.cc:290] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:33.748173 19322 raft_consensus.cc:357] T ffd063f8889b4bcdb9656466f0b144f3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.749037 19322 raft_consensus.cc:383] T ffd063f8889b4bcdb9656466f0b144f3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:33.749320 19322 raft_consensus.cc:738] T ffd063f8889b4bcdb9656466f0b144f3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:33.749989 19322 consensus_queue.cc:260] T ffd063f8889b4bcdb9656466f0b144f3 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.752043 19955 tablet_loader.cc:96] loaded metadata for tablet adf61f6cb63d4e8ea46b440e7ecce00d (table default.Table_c0ddb0fbf2f347c0b82acf9060722ed2 [id=964e4311f5ed4578a6f7285a9d2bbafa])
I20250111 02:57:33.756218 19322 ts_tablet_manager.cc:1428] T ffd063f8889b4bcdb9656466f0b144f3 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.023s user 0.004s sys 0.003s
I20250111 02:57:33.770205 19955 tablet_loader.cc:96] loaded metadata for tablet aec6dffdaa6b4798b7827a6a445601ca (table default.Table_0dcad07d56d843e59f351d933e625a21 [id=57e820dd99764a578d0c50f5f250aad4])
I20250111 02:57:33.772591 19322 tablet_bootstrap.cc:492] T 2e06e867266948df8eb35a85c58f578f P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:33.743031 18906 raft_consensus.cc:3054] T 3052e9707de14e1f9e38236d56908ed3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:33.799798 19955 tablet_loader.cc:96] loaded metadata for tablet b2f561cfd4f34426b7e42027e53b3ff1 (table default.Table_37a329a4875c463ea093ad5ab23e9cf6 [id=23a0f0c1aaa149c68390a0d0327ff2c5])
W20250111 02:57:33.809307 19863 log.cc:926] Time spent T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20250111 02:57:33.823873 19955 tablet_loader.cc:96] loaded metadata for tablet b4273afc264843029ed0baf1fc5c8a3e (table default.Table_4e3a79a5d2184687a8c51fd2702c5fd0 [id=73c59c0d9864497fbc9f19e265d8283f])
I20250111 02:57:33.825428 19955 tablet_loader.cc:96] loaded metadata for tablet b5148fa2e6504561bd1b1a19e48993d7 (table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0])
I20250111 02:57:33.744701 19330 tablet_bootstrap.cc:492] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
W20250111 02:57:33.778947 18989 leader_election.cc:343] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:33.830897 19322 tablet_bootstrap.cc:654] T 2e06e867266948df8eb35a85c58f578f P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:33.842048 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 63131.8us
I20250111 02:57:33.846221 19330 tablet_bootstrap.cc:654] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:33.846367 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46894710a72407993b36801f3b90eb0" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:33.847685 19049 raft_consensus.cc:2388] T e46894710a72407993b36801f3b90eb0 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate d0cedab82b154d62b78809edc6f28bfa in this term.
I20250111 02:57:33.849689 20055 consensus_queue.cc:1035] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:33.852712 19955 tablet_loader.cc:96] loaded metadata for tablet bc0bd71c6f4d4c90a4f3ea747af02165 (table default.Table_535081a0b0ff46998f523371377b75f9 [id=531ca04d38e9427eb955326135436b79])
I20250111 02:57:33.854089 19330 tablet_bootstrap.cc:492] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:33.870747 19955 tablet_loader.cc:96] loaded metadata for tablet c8f0c42e06d4428284fa006fe0062cf0 (table default.Table_6ad0b827920c42c68113310190019df1 [id=b25314ef99fc4e0ba80c6391223d2a3d])
W20250111 02:57:33.858353 18989 consensus_peers.cc:487] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:33.874519 18989 leader_election.cc:343] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:33.868824 19330 ts_tablet_manager.cc:1397] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.124s user 0.003s sys 0.010s
I20250111 02:57:33.879441 20122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:33.880263 20122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:33.882036 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e06e867266948df8eb35a85c58f578f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:33.882697 19193 raft_consensus.cc:2463] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:33.884267 19322 tablet_bootstrap.cc:492] T 2e06e867266948df8eb35a85c58f578f P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:33.885150 19322 ts_tablet_manager.cc:1397] T 2e06e867266948df8eb35a85c58f578f P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.113s user 0.008s sys 0.006s
I20250111 02:57:33.896746 19955 tablet_loader.cc:96] loaded metadata for tablet cb96635f30b34de48870daf5166813bc (table default.Table_39f0114c91ef4d1c86ef0cf522e31e11 [id=5189b95a9b0e4b47ac0d7768e08e902b])
I20250111 02:57:33.898067 19955 tablet_loader.cc:96] loaded metadata for tablet cc7158a7fc624fa4bc816ac3721c610d (table default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605])
I20250111 02:57:33.899258 20046 raft_consensus.cc:491] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:33.899695 20046 raft_consensus.cc:513] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.901546 20046 leader_election.cc:290] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:33.714778 19196 raft_consensus.cc:2463] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:33.904217 19955 tablet_loader.cc:96] loaded metadata for tablet d2fc07e0474e4fc5a46f2350229ca31f (table default.Table_6a0ed66485f640e9a2c11b62b5cbd239 [id=7b936eef4300409e908cc92b1453b500])
I20250111 02:57:33.905580 19955 tablet_loader.cc:96] loaded metadata for tablet d3b7616fe56f4e5fb49bdde6961fa008 (table default.Table_1ab2b1f61ced4c97ad5e44f18b6e8699 [id=40eb021e024747f0a173bf87538b7281])
I20250111 02:57:33.906284 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf61f6cb63d4e8ea46b440e7ecce00d" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:33.906916 19955 tablet_loader.cc:96] loaded metadata for tablet d6da575f052d4b20a8e41f0bd9dab7e4 (table default.Table_02b5991089a34b0b8ee6039ccc092c93 [id=977f59b134154bc396bf42bc1e44e717])
W20250111 02:57:33.917213 19131 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.20.124:34593, user_credentials={real_user=slave}} blocked reactor thread for 153776us
I20250111 02:57:33.924075 19955 tablet_loader.cc:96] loaded metadata for tablet d837bd001aa649e9b3ca94f7cdd9f0fe (table default.Table_b6421db891c148c688d25610559fe16f [id=190cfd157c8d4bd19a8f40cfd80925c1])
I20250111 02:57:33.925379 19955 tablet_loader.cc:96] loaded metadata for tablet d83cfd85a52f438dabe49b6223d5090f (table default.Table_799d0b8328de40a7bacb58ea433fec72 [id=e9efdc6c94574ddca092dd6b9f3b8419])
I20250111 02:57:33.925266 19330 raft_consensus.cc:357] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.925985 19330 raft_consensus.cc:383] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:33.926266 19330 raft_consensus.cc:738] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:33.926695 19955 tablet_loader.cc:96] loaded metadata for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe])
I20250111 02:57:33.927846 19955 tablet_loader.cc:96] loaded metadata for tablet dea4db154779483cb069118e3f3789b0 (table default.Table_8eae98e1e49c4f6c9e52644256812eaa [id=7dfae414add74b6491f2f7595a551342])
I20250111 02:57:33.929126 19955 tablet_loader.cc:96] loaded metadata for tablet df1870cceafe463b8d024e8b065850e0 (table default.Table_aea3cbca3247424c9a12e310135fe0b9 [id=c151ba8ccc454372b4ad7621b6bc7aeb])
I20250111 02:57:33.930524 19955 tablet_loader.cc:96] loaded metadata for tablet e01771c56ea5498a876d3bb865c752a2 (table default.Table_dacea9000ff840659052c41c94a8bfaf [id=51bd1567dc9c43b2877791b72c2c2046])
I20250111 02:57:33.939954 19955 tablet_loader.cc:96] loaded metadata for tablet e46894710a72407993b36801f3b90eb0 (table default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3])
I20250111 02:57:33.943857 18903 raft_consensus.cc:3054] T e46894710a72407993b36801f3b90eb0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:33.948669 19955 tablet_loader.cc:96] loaded metadata for tablet e4bd42ad66064bcaa043746081f1c759 (table default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7])
I20250111 02:57:33.949245 18624 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]). Expected version 2 got 0
I20250111 02:57:33.954643 19955 tablet_loader.cc:96] loaded metadata for tablet ea76b354a75041e888cf862250e4947b (table default.Table_b44f1ebbaf7045e2bcfefcee1c91c0d9 [id=ec5cf5f9b6234008914c0ed664260dcb])
I20250111 02:57:33.956189 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237ea33846044127987c1d81a3203553" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:33.956859 19194 raft_consensus.cc:2463] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:33.957513 18986 leader_election.cc:304] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:33.962046 18624 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet cc7158a7fc624fa4bc816ac3721c610d (table default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605]). Expected version 2 got 0
I20250111 02:57:33.921751 19321 tablet_bootstrap.cc:654] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:33.982383 19955 tablet_loader.cc:96] loaded metadata for tablet f1802e7349934fbb9d5f7eda37c3df20 (table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0])
I20250111 02:57:33.926862 19330 consensus_queue.cc:260] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.983844 19955 tablet_loader.cc:96] loaded metadata for tablet f31ad2e1895d4a26b8e9f7cd7366ef81 (table default.Table_23d632d4476c46b78d27619201589641 [id=db5ffbbe4fe7424f943fbaca17103d84])
I20250111 02:57:33.986209 20084 raft_consensus.cc:491] T 62c6b166a29943ceb15f093fac728c79 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250111 02:57:33.988643 18842 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: 32afc2f158254811836fcf00f7f409ed
I20250111 02:57:33.989073 18842 leader_election.cc:304] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:33.989014 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e06e867266948df8eb35a85c58f578f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:33.990319 18904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237ea33846044127987c1d81a3203553" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:33.991034 18904 raft_consensus.cc:2463] T 237ea33846044127987c1d81a3203553 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:33.986774 20084 raft_consensus.cc:513] T 62c6b166a29943ceb15f093fac728c79 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.997246 18624 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 7ea78feee64340e39404cba0f1bf04db (table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5]). Expected version 1 got 0
I20250111 02:57:33.998358 18624 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 855dac47872d4c5fbf8f5e38b588c7fe (table default.Table_49e4dd58d29f4cb6a0ed4f85138713b8 [id=f1373922978a4598b888e1863525942e]). Expected version 1 got 0
I20250111 02:57:33.998906 19322 raft_consensus.cc:357] T 2e06e867266948df8eb35a85c58f578f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:33.999688 19322 raft_consensus.cc:383] T 2e06e867266948df8eb35a85c58f578f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:33.999933 19322 raft_consensus.cc:738] T 2e06e867266948df8eb35a85c58f578f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:34.000593 20084 leader_election.cc:290] T 62c6b166a29943ceb15f093fac728c79 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:34.000473 19322 consensus_queue.cc:260] T 2e06e867266948df8eb35a85c58f578f P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.001194 20084 raft_consensus.cc:2743] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:34.003626 19322 ts_tablet_manager.cc:1428] T 2e06e867266948df8eb35a85c58f578f P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.118s user 0.010s sys 0.000s
I20250111 02:57:34.004624 19322 tablet_bootstrap.cc:492] T ea76b354a75041e888cf862250e4947b P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:34.013751 20055 raft_consensus.cc:2798] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:34.014245 20055 raft_consensus.cc:491] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:34.014640 20055 raft_consensus.cc:3054] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:34.017160 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c6b166a29943ceb15f093fac728c79" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:34.018168 19955 tablet_loader.cc:96] loaded metadata for tablet f778763e59a94969b3f62be87e2fd867 (table default.Table_902d917b19634ad5b20fadd455c0ba88 [id=ec311d46e9224fd38d1864b43717da4a])
I20250111 02:57:34.019415 19955 tablet_loader.cc:96] loaded metadata for tablet f9092a3453f94abbbf4bdd66a57d42cc (table default.Table_ee18021e6a544ed6b86b624aec8c39be [id=a9538f81833c4c0d8d02b6271a67f711])
I20250111 02:57:34.020046 20054 raft_consensus.cc:491] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:34.020516 20054 raft_consensus.cc:513] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.022507 20054 leader_election.cc:290] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:34.025041 20055 raft_consensus.cc:513] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.028333 20055 leader_election.cc:290] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:34.035746 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 76516.9us
I20250111 02:57:34.046175 19955 tablet_loader.cc:96] loaded metadata for tablet f9ec6b2a27744ee1aef0efdc98d8757c (table default.Table_f9d19027fe814825a6c3906baef77ff3 [id=9a4fbc1c2c2c44c182f9935156885c5a])
I20250111 02:57:34.047719 19955 tablet_loader.cc:96] loaded metadata for tablet fa86b1e6eca7403c9a9c52592b8fb636 (table default.Table_4b9e85e01e1843ce831205177cbb2e0f [id=0a118da8c5d0408980cbd0b33e0c2ed0])
I20250111 02:57:34.048883 19955 tablet_loader.cc:96] loaded metadata for tablet fb9b1c7e252d40e29e6d16311d145968 (table default.Table_e091dc5f47d9492aa86ff883e04fe415 [id=ee752760552944289c461f1c4c50d90f])
I20250111 02:57:34.050074 20059 raft_consensus.cc:491] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:34.050537 20059 raft_consensus.cc:513] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.045065 19322 tablet_bootstrap.cc:654] T ea76b354a75041e888cf862250e4947b P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:34.052444 20059 leader_election.cc:290] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:34.054119 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b6687f39d4246b49c0c3eb904b11c1a" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:34.055979 19133 leader_election.cc:343] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:34.058106 19955 tablet_loader.cc:96] loaded metadata for tablet ffd063f8889b4bcdb9656466f0b144f3 (table default.Table_53d44be8eaeb4e59a8e3454aac56daca [id=8de61815471842e593a7d29efffc443c])
I20250111 02:57:34.071444 20132 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:34.072077 20132 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:34.077922 19322 tablet_bootstrap.cc:492] T ea76b354a75041e888cf862250e4947b P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:34.078492 19322 ts_tablet_manager.cc:1397] T ea76b354a75041e888cf862250e4947b P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.074s user 0.008s sys 0.004s
I20250111 02:57:34.078709 19321 tablet_bootstrap.cc:492] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:34.079144 19321 ts_tablet_manager.cc:1397] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.340s user 0.014s sys 0.000s
I20250111 02:57:34.073158 20133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:34.081027 20133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:34.081194 19322 raft_consensus.cc:357] T ea76b354a75041e888cf862250e4947b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.081956 19322 raft_consensus.cc:383] T ea76b354a75041e888cf862250e4947b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:34.082235 19322 raft_consensus.cc:738] T ea76b354a75041e888cf862250e4947b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:33.999034 18624 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 9192160e931445f1acd207604192f759 (table default.Table_bccba04b48cc408d845624ee7ed98e5f [id=1496848ec9594cad889312978436d9c4]). Expected version 1 got 0
I20250111 02:57:34.082859 19322 consensus_queue.cc:260] T ea76b354a75041e888cf862250e4947b P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.084023 18624 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 845da3ee0863430ea3fe9de8c84f2c17 (table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3]). Expected version 2 got 0
I20250111 02:57:34.084760 18624 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]). Expected version 4 got 0
I20250111 02:57:34.085611 18624 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet a983a670c2104398ac1a159ac3a1da7c (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19]). Expected version 4 got 0
W20250111 02:57:34.028808 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 71384.4us
I20250111 02:57:34.030508 19330 ts_tablet_manager.cc:1428] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.154s user 0.007s sys 0.000s
W20250111 02:57:34.088434 19955 catalog_manager.cc:1477] Time spent T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c: Loading table and tablet metadata into memory: real 4.554s user 0.360s sys 0.048s
I20250111 02:57:34.088734 19955 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:57:34.088814 18986 leader_election.cc:304] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:34.089246 19330 tablet_bootstrap.cc:492] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:34.090503 19957 raft_consensus.cc:2798] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:34.090544 18986 leader_election.cc:304] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:34.091101 19957 raft_consensus.cc:695] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:34.091971 20046 raft_consensus.cc:2798] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:34.092407 20046 raft_consensus.cc:491] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:34.091948 19957 consensus_queue.cc:237] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.092761 20046 raft_consensus.cc:3054] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:34.094503 18626 catalog_manager.cc:5526] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: UNKNOWN } } }
I20250111 02:57:34.095953 18626 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet 7ea78feee64340e39404cba0f1bf04db (table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5]). Expected version 1 got 0
I20250111 02:57:34.096040 18906 raft_consensus.cc:2463] T 3052e9707de14e1f9e38236d56908ed3 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:34.097129 18626 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet a983a670c2104398ac1a159ac3a1da7c (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19]). Expected version 4 got 0
I20250111 02:57:34.097899 18626 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet cc7158a7fc624fa4bc816ac3721c610d (table default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605]). Expected version 2 got 0
W20250111 02:57:34.103457 18842 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 114851us
W20250111 02:57:34.103771 19131 leader_election.cc:343] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:34.104300 18842 leader_election.cc:304] T e46894710a72407993b36801f3b90eb0 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:34.104300 19131 leader_election.cc:304] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
W20250111 02:57:34.108938 18989 leader_election.cc:343] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:34.108759 18718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 4 candidate_status { last_received { term: 2 index: 277 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:34.109858 18718 raft_consensus.cc:2363] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8784b9603f8544b79373158aace063af for earlier term 4. Current term is 7.
W20250111 02:57:34.110420 18718 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36846 (request call id 302) took 4266 ms (client timeout 580 ms). Trace:
I20250111 02:57:34.112466 18639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 7 candidate_status { last_received { term: 6 index: 277 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:34.116837 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c6b166a29943ceb15f093fac728c79" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:34.118817 20132 leader_election.cc:290] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8784b9603f8544b79373158aace063af (127.18.20.125:40359), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:34.105746 20046 raft_consensus.cc:513] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.122817 19330 tablet_bootstrap.cc:654] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:34.112012 18709 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:58954 (request call id 3) took 2828 ms. Trace:
I20250111 02:57:34.124140 18717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 4 candidate_status { last_received { term: 2 index: 277 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:34.125170 18717 raft_consensus.cc:2363] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8784b9603f8544b79373158aace063af for earlier term 4. Current term is 7.
W20250111 02:57:34.125699 18717 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36846 (request call id 304) took 3676 ms (client timeout 580 ms). Trace:
W20250111 02:57:34.126135 18717 rpcz_store.cc:269] 0111 02:57:30.449488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:30.449714 (+ 226us) service_pool.cc:225] Handling call
0111 02:57:34.125597 (+3675883us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:34.128135 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 6 candidate_status { last_received { term: 5 index: 274 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:34.125932 19322 ts_tablet_manager.cc:1428] T ea76b354a75041e888cf862250e4947b P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.047s user 0.007s sys 0.000s
I20250111 02:57:34.129194 18713 raft_consensus.cc:2363] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e336d4ddf044bf5b0b6e4f5af634727 for earlier term 6. Current term is 7.
W20250111 02:57:34.129770 18713 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46906 (request call id 4) took 2472 ms (client timeout 520 ms). Trace:
W20250111 02:57:34.130273 18713 rpcz_store.cc:269] 0111 02:57:31.657689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:31.657946 (+ 257us) service_pool.cc:225] Handling call
0111 02:57:34.129662 (+2471716us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:34.130028 20046 leader_election.cc:290] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:34.130770 19322 tablet_bootstrap.cc:492] T 0ef86e1fd412484db83e7d39c9059a90 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:34.133780 20135 raft_consensus.cc:2743] T e46894710a72407993b36801f3b90eb0 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:34.131932 18712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 6 candidate_status { last_received { term: 5 index: 274 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:34.136535 18712 raft_consensus.cc:2363] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8784b9603f8544b79373158aace063af for earlier term 6. Current term is 7.
I20250111 02:57:34.123268 18709 rpcz_store.cc:276] 0111 02:57:31.283411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:31.283577 (+ 166us) service_pool.cc:225] Handling call
0111 02:57:34.111587 (+2828010us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:34.110797 18718 rpcz_store.cc:269] 0111 02:57:29.844356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:29.844479 (+ 123us) service_pool.cc:225] Handling call
0111 02:57:34.110302 (+4265823us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:34.137220 18712 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36846 (request call id 307) took 2040 ms (client timeout 665 ms). Trace:
I20250111 02:57:34.165452 20059 raft_consensus.cc:2743] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:34.176034 18902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef86e1fd412484db83e7d39c9059a90" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:34.178668 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e06e867266948df8eb35a85c58f578f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:34.179136 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237ea33846044127987c1d81a3203553" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:34.179406 18906 raft_consensus.cc:3054] T 2e06e867266948df8eb35a85c58f578f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:34.179777 18901 raft_consensus.cc:3054] T 237ea33846044127987c1d81a3203553 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:34.184110 19321 raft_consensus.cc:357] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.184473 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef86e1fd412484db83e7d39c9059a90" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:34.184816 19321 raft_consensus.cc:383] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:34.185074 19321 raft_consensus.cc:738] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:34.185216 19193 raft_consensus.cc:2463] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:34.185619 19321 consensus_queue.cc:260] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.187928 19321 ts_tablet_manager.cc:1428] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.109s user 0.006s sys 0.000s
I20250111 02:57:34.188107 19322 tablet_bootstrap.cc:654] T 0ef86e1fd412484db83e7d39c9059a90 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:34.188902 19321 tablet_bootstrap.cc:492] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:34.189256 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e06e867266948df8eb35a85c58f578f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:34.189942 19195 raft_consensus.cc:3054] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:34.204030 19330 tablet_bootstrap.cc:492] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:34.204459 19330 ts_tablet_manager.cc:1397] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.115s user 0.011s sys 0.002s
W20250111 02:57:34.211165 18986 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 57324.4us
I20250111 02:57:34.214576 18986 leader_election.cc:304] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:34.149562 18715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 4 candidate_status { last_received { term: 2 index: 277 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c"
I20250111 02:57:34.218238 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b6687f39d4246b49c0c3eb904b11c1a" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:34.219347 18901 raft_consensus.cc:2463] T 237ea33846044127987c1d81a3203553 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
W20250111 02:57:34.223214 19131 leader_election.cc:343] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:34.223685 19131 leader_election.cc:304] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:34.215374 19957 raft_consensus.cc:2798] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:34.229290 19957 raft_consensus.cc:491] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:34.229739 19957 raft_consensus.cc:3054] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:34.211655 18711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 6 candidate_status { last_received { term: 5 index: 274 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
W20250111 02:57:34.239645 18989 leader_election.cc:343] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:34.212409 18712 rpcz_store.cc:269] 0111 02:57:32.097001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:32.097257 (+ 256us) service_pool.cc:225] Handling call
0111 02:57:34.137111 (+2039854us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:34.247145 18718 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36846 (request call id 309) took 1444 ms (client timeout 665 ms). Trace:
W20250111 02:57:34.247634 18718 rpcz_store.cc:269] 0111 02:57:32.800541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:34.222409 (+1421868us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:34.244856 (+ 22447us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250111 02:57:34.254801 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 4 candidate_status { last_received { term: 2 index: 277 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c"
W20250111 02:57:34.256074 18719 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36846 (request call id 303) took 4345 ms (client timeout 594 ms). Trace:
W20250111 02:57:34.256420 18719 rpcz_store.cc:269] 0111 02:57:29.910550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:29.910687 (+ 137us) service_pool.cc:225] Handling call
0111 02:57:34.255992 (+4345305us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:34.244403 18716 raft_consensus.cc:1235] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 7 FOLLOWER]: Rejecting Update request from peer 8784b9603f8544b79373158aace063af for earlier term 4. Current term is 7. Ops: []
I20250111 02:57:34.258562 18716 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36846 (request call id 305) took 3765 ms. Trace:
I20250111 02:57:34.258997 18716 rpcz_store.cc:276] 0111 02:57:30.493076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:30.493286 (+ 210us) service_pool.cc:225] Handling call
0111 02:57:34.225043 (+3731757us) raft_consensus.cc:1466] Updating replica for 0 ops
0111 02:57:34.258140 (+ 33097us) raft_consensus.cc:1692] Filling consensus response to leader.
0111 02:57:34.258436 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:34.260548 18714 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:46906 (request call id 3) took 2688 ms. Trace:
I20250111 02:57:34.260721 18622 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927] alter complete (version 1)
I20250111 02:57:34.261003 18714 rpcz_store.cc:276] 0111 02:57:31.571956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:31.572187 (+ 231us) service_pool.cc:225] Handling call
0111 02:57:34.260396 (+2688209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:34.238878 18710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 6 candidate_status { last_received { term: 5 index: 274 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c"
I20250111 02:57:34.262668 18710 raft_consensus.cc:2363] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8784b9603f8544b79373158aace063af for earlier term 6. Current term is 7.
W20250111 02:57:34.263804 18710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36846 (request call id 308) took 1942 ms (client timeout 614 ms). Trace:
W20250111 02:57:34.264247 18710 rpcz_store.cc:269] 0111 02:57:32.321827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:32.322116 (+ 289us) service_pool.cc:225] Handling call
0111 02:57:34.263697 (+1941581us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:34.272136 18710 raft_consensus.cc:1235] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 7 FOLLOWER]: Rejecting Update request from peer 8784b9603f8544b79373158aace063af for earlier term 6. Current term is 7. Ops: []
I20250111 02:57:34.272984 18710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36846 (request call id 310) took 1360 ms. Trace:
I20250111 02:57:34.273515 18710 rpcz_store.cc:276] 0111 02:57:32.912431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:34.271568 (+1359137us) service_pool.cc:225] Handling call
0111 02:57:34.271926 (+ 358us) raft_consensus.cc:1466] Updating replica for 0 ops
0111 02:57:34.272596 (+ 670us) raft_consensus.cc:1692] Filling consensus response to leader.
0111 02:57:34.272870 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:34.275146 18711 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46906 (request call id 5) took 1972 ms (client timeout 520 ms). Trace:
W20250111 02:57:34.275648 18711 rpcz_store.cc:269] 0111 02:57:32.302238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:32.302506 (+ 268us) service_pool.cc:225] Handling call
0111 02:57:34.274994 (+1972488us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:34.277169 18715 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36846 (request call id 306) took 3766 ms (client timeout 594 ms). Trace:
W20250111 02:57:34.277649 18715 rpcz_store.cc:269] 0111 02:57:30.509583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:30.509812 (+ 229us) service_pool.cc:225] Handling call
0111 02:57:34.247080 (+3737268us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:34.291230 20122 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 }, Status: INVALID_TERM, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.001s
I20250111 02:57:34.292214 20122 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 6 LEADER]: Stepping down as leader of term 6
I20250111 02:57:34.292599 20122 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8784b9603f8544b79373158aace063af, State: Running, Role: LEADER
I20250111 02:57:34.293537 20122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 6.279, Last appended by leader: 279, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:34.294878 20122 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 6 FOLLOWER]: Advancing to term 7
W20250111 02:57:34.248349 18713 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36846 (request call id 311) took 1306 ms (client timeout 614 ms). Trace:
W20250111 02:57:34.319408 18713 rpcz_store.cc:269] 0111 02:57:32.938381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:34.224696 (+1286315us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:34.244921 (+ 20225us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250111 02:57:34.276563 18717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 7 candidate_status { last_received { term: 6 index: 277 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:34.321030 18717 raft_consensus.cc:2388] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e336d4ddf044bf5b0b6e4f5af634727 in current term 7: Already voted for candidate c34b72b7c70343d3b6f3fae79ce4be6c in this term.
I20250111 02:57:34.327790 19595 leader_election.cc:304] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727; no voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:34.329643 20132 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 6 FOLLOWER]: Advancing to term 7
I20250111 02:57:34.337575 20132 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250111 02:57:34.337493 18624 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038], current schema version=4): Not found: failed to reset TS proxy: No leader found: Tablet 6a03880a8b0e46b7858ed1c2c2dddcaa consensus state has no leader
W20250111 02:57:34.341982 18624 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.67:53171 (request call id 41) took 6382 ms (client timeout 1000 ms). Trace:
W20250111 02:57:34.349013 18624 rpcz_store.cc:269] 0111 02:57:27.960141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:28.687790 (+727649us) service_pool.cc:225] Handling call
0111 02:57:34.341883 (+5654093us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70451,"reported_tablets":24}
I20250111 02:57:34.369387 20143 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:34.370009 20143 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:34.382395 19321 tablet_bootstrap.cc:654] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:34.383479 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 95964.8us
I20250111 02:57:34.388654 18639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 8 candidate_status { last_received { term: 5 index: 274 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:34.389544 18639 raft_consensus.cc:2405] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c34b72b7c70343d3b6f3fae79ce4be6c for term 8 because replica has last-logged OpId of term: 6 index: 279, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 274.
I20250111 02:57:34.374437 19955 catalog_manager.cc:1260] Loaded cluster ID: 505f17df54394056975cd03f4ae103e8
I20250111 02:57:34.395303 19955 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:57:34.400408 19330 raft_consensus.cc:357] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.401449 19330 raft_consensus.cc:383] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:34.401789 19330 raft_consensus.cc:738] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:34.247627 18842 leader_election.cc:304] T 62c6b166a29943ceb15f093fac728c79 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:34.407312 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237ea33846044127987c1d81a3203553" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
W20250111 02:57:34.407507 18842 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 159997us
I20250111 02:57:34.409157 19957 raft_consensus.cc:513] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.402611 19330 consensus_queue.cc:260] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.411288 19957 leader_election.cc:290] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:34.414790 20046 consensus_queue.cc:1035] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:34.418795 20054 consensus_queue.cc:1035] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250111 02:57:34.408022 19192 raft_consensus.cc:3054] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:34.427567 19330 ts_tablet_manager.cc:1428] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.223s user 0.009s sys 0.000s
I20250111 02:57:34.408213 20059 raft_consensus.cc:2743] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:34.429416 19330 tablet_bootstrap.cc:492] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:34.419639 20135 raft_consensus.cc:2743] T 62c6b166a29943ceb15f093fac728c79 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:34.447335 19322 tablet_bootstrap.cc:492] T 0ef86e1fd412484db83e7d39c9059a90 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:34.447322 18903 raft_consensus.cc:3054] T ffd063f8889b4bcdb9656466f0b144f3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:34.447824 19322 ts_tablet_manager.cc:1397] T 0ef86e1fd412484db83e7d39c9059a90 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.317s user 0.042s sys 0.054s
I20250111 02:57:34.450130 19322 raft_consensus.cc:357] T 0ef86e1fd412484db83e7d39c9059a90 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.450829 19322 raft_consensus.cc:383] T 0ef86e1fd412484db83e7d39c9059a90 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:34.450690 18902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef86e1fd412484db83e7d39c9059a90" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:34.454450 18989 leader_election.cc:304] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:34.457051 19322 raft_consensus.cc:738] T 0ef86e1fd412484db83e7d39c9059a90 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:34.457784 19322 consensus_queue.cc:260] T 0ef86e1fd412484db83e7d39c9059a90 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.462345 19322 ts_tablet_manager.cc:1428] T 0ef86e1fd412484db83e7d39c9059a90 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250111 02:57:34.463371 19322 tablet_bootstrap.cc:492] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:34.459179 18902 raft_consensus.cc:3054] T 0ef86e1fd412484db83e7d39c9059a90 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:34.490643 19330 tablet_bootstrap.cc:654] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:34.493809 18906 raft_consensus.cc:2463] T 2e06e867266948df8eb35a85c58f578f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
W20250111 02:57:34.498420 18615 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6a03880a8b0e46b7858ed1c2c2dddcaa consensus state has no leader
I20250111 02:57:34.494011 19322 tablet_bootstrap.cc:654] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:34.499660 20054 raft_consensus.cc:491] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250111 02:57:34.499648 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 45269.9us
I20250111 02:57:34.500166 20054 raft_consensus.cc:513] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.502611 20054 leader_election.cc:290] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:34.511430 20054 raft_consensus.cc:2798] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:34.511924 20054 raft_consensus.cc:695] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:34.512846 20054 consensus_queue.cc:237] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.526299 19195 raft_consensus.cc:2463] T 2e06e867266948df8eb35a85c58f578f P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:34.433434 20131 consensus_queue.cc:1035] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.093s
I20250111 02:57:34.527257 19754 consensus_queue.cc:1035] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:34.533562 19955 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:57:34.534440 20135 raft_consensus.cc:491] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:34.534574 20136 raft_consensus.cc:491] T ea76b354a75041e888cf862250e4947b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:34.534927 20135 raft_consensus.cc:513] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:34.535019 20136 raft_consensus.cc:513] T ea76b354a75041e888cf862250e4947b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.536942 20135 leader_election.cc:290] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:34.537026 18989 leader_election.cc:304] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:34.537791 18904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf61f6cb63d4e8ea46b440e7ecce00d" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:34.536957 20136 leader_election.cc:290] T ea76b354a75041e888cf862250e4947b P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:34.539474 20054 raft_consensus.cc:2798] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:34.539944 20054 raft_consensus.cc:695] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:34.540735 20054 consensus_queue.cc:237] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.549302 18902 raft_consensus.cc:2463] T 0ef86e1fd412484db83e7d39c9059a90 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:34.401654 20143 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
W20250111 02:57:34.554438 18989 leader_election.cc:343] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:34.555207 19309 master_runner.cc:232] Existing masters have differing terms: 5 vs 7
I20250111 02:57:34.556326 19309 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:57:34.557567 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bd42ad66064bcaa043746081f1c759" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:34.558225 19049 raft_consensus.cc:2463] T e4bd42ad66064bcaa043746081f1c759 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:34.569499 19330 tablet_bootstrap.cc:492] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:34.569963 19330 ts_tablet_manager.cc:1397] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.142s user 0.009s sys 0.000s
I20250111 02:57:34.572259 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea76b354a75041e888cf862250e4947b" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:34.577105 18842 leader_election.cc:304] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:34.577966 20136 raft_consensus.cc:2798] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:34.578364 20136 raft_consensus.cc:491] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:34.578747 20136 raft_consensus.cc:3054] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:34.585763 20136 raft_consensus.cc:513] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:34.590008 19173 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf1ce65ccf54358b95665b9764fbdb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:34.590479 19192 raft_consensus.cc:2463] T 237ea33846044127987c1d81a3203553 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:34.596947 19169 tablet_service.cc:1514] Processing DeleteTablet for tablet 32afc2f158254811836fcf00f7f409ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:19 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:34.597820 19176 tablet_service.cc:1514] Processing DeleteTablet for tablet 92aa64b2de5746d1b12cc93a10c9d56e with delete_type TABLET_DATA_DELETED (replaced by tablet e46894710a72407993b36801f3b90eb0) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:34.598589 19165 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:34.599395 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2377893bfc426fa00d5ddacd9c5961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:21 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:34.600791 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 0267a92995ea460786dc26813b5895cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:22 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:34.603610 19171 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cd364f9a6894d19826deb59a8d8a612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:34.605824 19165 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b302fcba27244c193586bc25fc9305f with delete_type TABLET_DATA_DELETED (replaced by tablet 13f0a0e7698e49a5a57562dc8ad52cd5) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:34.609622 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef86e1fd412484db83e7d39c9059a90" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:34.610278 19196 raft_consensus.cc:3054] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:34.618602 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 8 candidate_status { last_received { term: 5 index: 274 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:34.620966 18695 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:34.625572 20143 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 02:57:34.634441 19330 raft_consensus.cc:357] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.653834 19321 tablet_bootstrap.cc:492] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:34.655534 19321 ts_tablet_manager.cc:1397] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.467s user 0.011s sys 0.017s
I20250111 02:57:34.635176 19330 raft_consensus.cc:383] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:34.655822 19330 raft_consensus.cc:738] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:34.657382 19330 consensus_queue.cc:260] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.659581 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf61f6cb63d4e8ea46b440e7ecce00d" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:34.660276 19195 raft_consensus.cc:2463] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
W20250111 02:57:34.668174 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 5cf1ce65ccf54358b95665b9764fbdb0 because the tablet was not found. No further retry: Not found: Tablet not found: 5cf1ce65ccf54358b95665b9764fbdb0
W20250111 02:57:34.670164 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 32afc2f158254811836fcf00f7f409ed because the tablet was not found. No further retry: Not found: Tablet not found: 32afc2f158254811836fcf00f7f409ed
W20250111 02:57:34.672829 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 92aa64b2de5746d1b12cc93a10c9d56e because the tablet was not found. No further retry: Not found: Tablet not found: 92aa64b2de5746d1b12cc93a10c9d56e
W20250111 02:57:34.673795 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 because the tablet was not found. No further retry: Not found: Tablet not found: 5fe8aa967fcb4bd7a48f6f0f69bea496
W20250111 02:57:34.674736 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 1b2377893bfc426fa00d5ddacd9c5961 because the tablet was not found. No further retry: Not found: Tablet not found: 1b2377893bfc426fa00d5ddacd9c5961
I20250111 02:57:34.676662 18986 leader_election.cc:304] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:34.677793 20149 raft_consensus.cc:2798] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:34.678205 20149 raft_consensus.cc:491] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:34.678628 20149 raft_consensus.cc:3054] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:34.683135 19330 ts_tablet_manager.cc:1428] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.113s user 0.005s sys 0.002s
I20250111 02:57:34.711417 20122 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:34.711920 20122 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:34.713904 20122 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:34.715515 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 8 candidate_status { last_received { term: 6 index: 279 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:34.715981 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 8 candidate_status { last_received { term: 6 index: 279 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:34.716158 18713 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 7.
I20250111 02:57:34.717577 18615 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c; no voters:
I20250111 02:57:34.719120 20122 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250111 02:57:34.719448 20122 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:34.719781 20122 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 7 FOLLOWER]: Advancing to term 8
I20250111 02:57:34.724581 20155 tablet_replica.cc:331] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:34.725646 20155 raft_consensus.cc:2238] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:34.726284 20155 raft_consensus.cc:2267] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:34.728413 20155 ts_tablet_manager.cc:1905] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:34.734275 19330 tablet_bootstrap.cc:492] T 933828fd6fb041e9a4889f4ea8a3c5ff P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
W20250111 02:57:34.740252 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 0267a92995ea460786dc26813b5895cf because the tablet was not found. No further retry: Not found: Tablet not found: 0267a92995ea460786dc26813b5895cf
I20250111 02:57:34.753991 19330 tablet_bootstrap.cc:654] T 933828fd6fb041e9a4889f4ea8a3c5ff P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:34.734982 19196 raft_consensus.cc:2463] T 0ef86e1fd412484db83e7d39c9059a90 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:34.736658 19321 raft_consensus.cc:357] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.775439 19321 raft_consensus.cc:383] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:34.775833 19321 raft_consensus.cc:738] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:34.776602 19321 consensus_queue.cc:260] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.779304 19321 ts_tablet_manager.cc:1428] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.123s user 0.005s sys 0.002s
I20250111 02:57:34.780283 19321 tablet_bootstrap.cc:492] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:34.783020 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea76b354a75041e888cf862250e4947b" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:34.783995 19192 raft_consensus.cc:2388] T ea76b354a75041e888cf862250e4947b P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:34.790135 20136 leader_election.cc:290] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Requested vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:34.777160 20122 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:34.791785 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bd42ad66064bcaa043746081f1c759" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:34.792515 19196 raft_consensus.cc:2463] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:34.798951 19321 tablet_bootstrap.cc:654] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:34.810801 19955 catalog_manager.cc:5910] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c: Loaded TSK: 0
I20250111 02:57:34.827188 19955 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:57:34.831276 20151 raft_consensus.cc:491] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:34.831806 20151 raft_consensus.cc:513] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.833786 20151 leader_election.cc:290] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:34.841209 20155 ts_tablet_manager.cc:1918] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:34.866995 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 8 candidate_status { last_received { term: 6 index: 279 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727"
I20250111 02:57:34.867909 19683 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 7 FOLLOWER]: Advancing to term 8
I20250111 02:57:34.730432 19322 tablet_bootstrap.cc:492] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:34.871007 19322 ts_tablet_manager.cc:1397] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.408s user 0.014s sys 0.007s
I20250111 02:57:34.856851 19321 tablet_bootstrap.cc:492] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:34.871940 19321 ts_tablet_manager.cc:1397] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.092s user 0.011s sys 0.000s
I20250111 02:57:34.873518 19322 raft_consensus.cc:357] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.874222 19322 raft_consensus.cc:383] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:34.874483 19322 raft_consensus.cc:738] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:34.875103 19322 consensus_queue.cc:260] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:34.877269 19322 ts_tablet_manager.cc:1428] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250111 02:57:34.878204 20122 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 8 election: Requested vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:34.882746 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 8 candidate_status { last_received { term: 6 index: 279 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c"
I20250111 02:57:34.883391 18713 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 7 FOLLOWER]: Advancing to term 8
I20250111 02:57:34.888073 19322 tablet_bootstrap.cc:492] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:34.895431 18713 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 8.
I20250111 02:57:34.898905 20155 log.cc:1198] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/2cd364f9a6894d19826deb59a8d8a612
I20250111 02:57:34.899910 20155 ts_tablet_manager.cc:1939] T 2cd364f9a6894d19826deb59a8d8a612 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:34.900488 18615 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c; no voters:
I20250111 02:57:34.901387 20122 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 8 FOLLOWER]: Leader election won for term 8
I20250111 02:57:34.912212 20122 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 8 LEADER]: Becoming Leader. State: Replica: 8784b9603f8544b79373158aace063af, State: Running, Role: LEADER
I20250111 02:57:34.913257 20122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 6.279, Last appended by leader: 279, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:34.918426 19683 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 8.
I20250111 02:57:34.920938 20164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8784b9603f8544b79373158aace063af. Latest consensus state: current_term: 8 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
W20250111 02:57:34.928915 19955 catalog_manager.cc:1410] Failed waiting for node to catch up after master election: Illegal state: Node c34b72b7c70343d3b6f3fae79ce4be6c not leader. Raft Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
W20250111 02:57:34.929903 19955 catalog_manager.cc:1410] Failed waiting for node to catch up after master election: Illegal state: Node c34b72b7c70343d3b6f3fae79ce4be6c not leader. Raft Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
W20250111 02:57:34.707161 18842 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 130151us
W20250111 02:57:34.930701 19955 catalog_manager.cc:1410] Failed waiting for node to catch up after master election: Illegal state: Node c34b72b7c70343d3b6f3fae79ce4be6c not leader. Raft Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:34.931411 18842 leader_election.cc:304] T ea76b354a75041e888cf862250e4947b P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:34.931895 20164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:34.943765 20136 raft_consensus.cc:3054] T ea76b354a75041e888cf862250e4947b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:34.956379 18989 leader_election.cc:304] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:34.966881 20136 raft_consensus.cc:2743] T ea76b354a75041e888cf862250e4947b P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:34.970934 20046 raft_consensus.cc:2798] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:34.971602 20046 raft_consensus.cc:695] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:34.972530 20046 consensus_queue.cc:237] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.007468 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bd42ad66064bcaa043746081f1c759" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:35.008167 19049 raft_consensus.cc:3054] T e4bd42ad66064bcaa043746081f1c759 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:35.042984 20136 raft_consensus.cc:491] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:35.043565 20136 raft_consensus.cc:513] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.045567 20136 leader_election.cc:290] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:35.046353 19133 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 184052us
W20250111 02:57:35.052336 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 68706.2us
I20250111 02:57:35.056260 19049 raft_consensus.cc:2463] T e4bd42ad66064bcaa043746081f1c759 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:35.057906 19321 raft_consensus.cc:357] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.058677 19321 raft_consensus.cc:383] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:35.058957 19321 raft_consensus.cc:738] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:35.059628 19321 consensus_queue.cc:260] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.061590 19321 ts_tablet_manager.cc:1428] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.189s user 0.005s sys 0.001s
I20250111 02:57:35.062788 19321 tablet_bootstrap.cc:492] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:35.063977 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 2cd364f9a6894d19826deb59a8d8a612 (table default.Table_71ceb23d653647089e7bd2fa3da23c01 [id=7e3413aa9c8d4037928c497a8afad14e]) successfully deleted
W20250111 02:57:35.064666 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 4b302fcba27244c193586bc25fc9305f because the tablet was not found. No further retry: Not found: Tablet not found: 4b302fcba27244c193586bc25fc9305f
I20250111 02:57:35.072597 19321 tablet_bootstrap.cc:654] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:35.087508 20131 raft_consensus.cc:491] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:35.088029 20131 raft_consensus.cc:513] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.089949 20131 leader_election.cc:290] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:35.093022 19321 tablet_bootstrap.cc:492] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:35.093449 19321 ts_tablet_manager.cc:1397] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.002s
I20250111 02:57:35.095798 19321 raft_consensus.cc:357] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.096483 19321 raft_consensus.cc:383] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:35.096741 19321 raft_consensus.cc:738] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:35.097326 19321 consensus_queue.cc:260] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.099022 19321 ts_tablet_manager.cc:1428] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20250111 02:57:35.113938 19321 tablet_bootstrap.cc:492] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:35.120026 19321 tablet_bootstrap.cc:654] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:35.123950 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb96635f30b34de48870daf5166813bc" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:35.124655 19192 raft_consensus.cc:2463] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:35.127740 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4bd42ad66064bcaa043746081f1c759" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:35.128466 19195 raft_consensus.cc:3054] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:35.136880 19322 tablet_bootstrap.cc:654] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:35.057127 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4273afc264843029ed0baf1fc5c8a3e" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:35.143620 19321 tablet_bootstrap.cc:492] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:35.144055 19321 ts_tablet_manager.cc:1397] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.000s
I20250111 02:57:35.100566 19330 tablet_bootstrap.cc:492] T 933828fd6fb041e9a4889f4ea8a3c5ff P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:35.115662 20054 consensus_queue.cc:1035] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:35.144703 19330 ts_tablet_manager.cc:1397] T 933828fd6fb041e9a4889f4ea8a3c5ff P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.411s user 0.012s sys 0.009s
I20250111 02:57:35.116276 19957 consensus_queue.cc:1035] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:35.123047 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb96635f30b34de48870daf5166813bc" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:35.146229 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4273afc264843029ed0baf1fc5c8a3e" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:35.139238 20046 raft_consensus.cc:491] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:35.147289 20046 raft_consensus.cc:513] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.149210 20046 leader_election.cc:290] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:35.149760 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 91389.2us
W20250111 02:57:35.150940 18986 consensus_peers.cc:487] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cd364f9a6894d19826deb59a8d8a612. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:35.154119 19923 consensus_queue.cc:1035] T 3052e9707de14e1f9e38236d56908ed3 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250111 02:57:35.157660 19131 leader_election.cc:343] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:35.147805 19321 raft_consensus.cc:357] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.160662 19330 raft_consensus.cc:357] T 933828fd6fb041e9a4889f4ea8a3c5ff P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.161237 19321 raft_consensus.cc:383] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:35.161430 19330 raft_consensus.cc:383] T 933828fd6fb041e9a4889f4ea8a3c5ff P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:35.161515 19321 raft_consensus.cc:738] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:35.161674 19330 raft_consensus.cc:738] T 933828fd6fb041e9a4889f4ea8a3c5ff P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:35.162097 19321 consensus_queue.cc:260] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.162387 19330 consensus_queue.cc:260] T 933828fd6fb041e9a4889f4ea8a3c5ff P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.165659 19330 ts_tablet_manager.cc:1428] T 933828fd6fb041e9a4889f4ea8a3c5ff P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20250111 02:57:35.172369 19321 ts_tablet_manager.cc:1428] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250111 02:57:35.176607 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933828fd6fb041e9a4889f4ea8a3c5ff" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:35.178870 19133 leader_election.cc:343] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:35.179474 19133 leader_election.cc:304] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:35.180249 20151 raft_consensus.cc:2743] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:35.183367 18841 leader_election.cc:304] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:35.184224 20136 raft_consensus.cc:2798] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:35.184566 20136 raft_consensus.cc:491] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:35.184880 20136 raft_consensus.cc:3054] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:35.208042 19923 raft_consensus.cc:491] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:35.208667 19923 raft_consensus.cc:513] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.211014 19923 leader_election.cc:290] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:35.215317 18902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "061e6eb5775443679e9cbd6b11f66fe3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:35.221550 19131 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.20.124:34593, user_credentials={real_user=slave}} blocked reactor thread for 52273.5us
I20250111 02:57:35.142326 20149 raft_consensus.cc:513] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.229097 20149 leader_election.cc:290] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:35.219026 18842 leader_election.cc:304] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:35.231531 20135 raft_consensus.cc:2798] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:35.233379 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "061e6eb5775443679e9cbd6b11f66fe3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:35.233881 19321 tablet_bootstrap.cc:492] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:35.234359 19192 raft_consensus.cc:2463] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
W20250111 02:57:35.240872 18842 leader_election.cc:343] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:35.241279 19321 tablet_bootstrap.cc:654] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:35.242497 19133 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 63677.9us
W20250111 02:57:35.246007 19133 leader_election.cc:343] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:35.249940 18989 leader_election.cc:343] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:35.244385 20136 raft_consensus.cc:513] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.263788 20136 leader_election.cc:290] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:35.264317 19321 tablet_bootstrap.cc:492] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:35.264732 19321 ts_tablet_manager.cc:1397] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.004s
I20250111 02:57:35.267438 19321 raft_consensus.cc:357] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.268105 19321 raft_consensus.cc:383] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:35.268343 19321 raft_consensus.cc:738] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:35.269793 18904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf61f6cb63d4e8ea46b440e7ecce00d" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:35.268893 19321 consensus_queue.cc:260] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.272069 18902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26e5db463e94d919640a07a2f8574a8" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:35.278724 20135 raft_consensus.cc:695] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 LEADER]: Becoming Leader. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Running, Role: LEADER
I20250111 02:57:35.278982 19321 ts_tablet_manager.cc:1428] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.014s user 0.003s sys 0.002s
I20250111 02:57:35.275712 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26e5db463e94d919640a07a2f8574a8" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:35.281567 19321 tablet_bootstrap.cc:492] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:35.281634 19192 raft_consensus.cc:2463] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:35.279914 20135 consensus_queue.cc:237] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:35.272497 19195 raft_consensus.cc:2463] T e4bd42ad66064bcaa043746081f1c759 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:35.297372 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb96635f30b34de48870daf5166813bc" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:35.298107 19194 raft_consensus.cc:3054] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:35.310490 19321 tablet_bootstrap.cc:654] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:35.312515 19322 tablet_bootstrap.cc:492] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:35.313041 19322 ts_tablet_manager.cc:1397] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.425s user 0.009s sys 0.005s
I20250111 02:57:35.322172 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf61f6cb63d4e8ea46b440e7ecce00d" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:35.322921 19192 raft_consensus.cc:3054] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:35.326668 19683 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Refusing update from remote peer 8784b9603f8544b79373158aace063af: Log matching property violated. Preceding OpId in replica: term: 6 index: 278. Preceding OpId from leader: term: 8 index: 280. (index mismatch)
I20250111 02:57:35.328397 20164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 277, Time since last communication: 0.001s
I20250111 02:57:35.329869 18986 leader_election.cc:304] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:35.334003 19321 tablet_bootstrap.cc:492] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:35.334447 19321 ts_tablet_manager.cc:1397] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.009s
I20250111 02:57:35.337850 19321 raft_consensus.cc:357] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.338533 19321 raft_consensus.cc:383] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:35.338786 19321 raft_consensus.cc:738] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:35.339633 20046 raft_consensus.cc:2798] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:35.339464 19321 consensus_queue.cc:260] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.340204 20046 raft_consensus.cc:491] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:35.340689 20046 raft_consensus.cc:3054] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:35.342123 19321 ts_tablet_manager.cc:1428] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250111 02:57:35.343016 19321 tablet_bootstrap.cc:492] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:35.349864 19321 tablet_bootstrap.cc:654] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:35.352705 19194 raft_consensus.cc:2463] T cb96635f30b34de48870daf5166813bc P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
W20250111 02:57:35.354780 18989 leader_election.cc:343] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:35.355588 18989 leader_election.cc:343] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:35.364675 18841 leader_election.cc:304] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:35.365803 20135 raft_consensus.cc:2798] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:35.366509 20135 raft_consensus.cc:695] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [term 1 LEADER]: Becoming Leader. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Running, Role: LEADER
I20250111 02:57:35.368567 20135 consensus_queue.cc:237] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.378329 19321 tablet_bootstrap.cc:492] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:35.378777 19321 ts_tablet_manager.cc:1397] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.007s
I20250111 02:57:35.380468 19192 raft_consensus.cc:2463] T adf61f6cb63d4e8ea46b440e7ecce00d P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:35.381592 19321 raft_consensus.cc:357] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.382278 19321 raft_consensus.cc:383] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:35.382539 19321 raft_consensus.cc:738] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:35.382264 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933828fd6fb041e9a4889f4ea8a3c5ff" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:35.383024 19050 raft_consensus.cc:2463] T 933828fd6fb041e9a4889f4ea8a3c5ff P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:35.383184 19321 consensus_queue.cc:260] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.384238 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb96635f30b34de48870daf5166813bc" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:35.385417 19321 ts_tablet_manager.cc:1428] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250111 02:57:35.386329 19321 tablet_bootstrap.cc:492] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
W20250111 02:57:35.383030 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 53278.7us
I20250111 02:57:35.388643 18986 leader_election.cc:304] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:35.389828 20054 raft_consensus.cc:2798] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:35.390074 18986 leader_election.cc:304] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:35.390228 20054 raft_consensus.cc:491] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:35.390645 20054 raft_consensus.cc:3054] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:35.392304 19923 raft_consensus.cc:2798] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:35.392776 19923 raft_consensus.cc:695] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:35.393843 19923 consensus_queue.cc:237] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.402904 20054 raft_consensus.cc:513] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.405004 20054 leader_election.cc:290] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:35.400173 19322 raft_consensus.cc:357] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.407557 19322 raft_consensus.cc:383] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:35.407860 19322 raft_consensus.cc:738] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:35.408486 19322 consensus_queue.cc:260] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.413242 20046 raft_consensus.cc:513] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.422498 20151 raft_consensus.cc:491] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:35.423010 20151 raft_consensus.cc:513] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.424729 19330 tablet_bootstrap.cc:492] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:35.425326 20151 leader_election.cc:290] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:35.431412 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7145efe5c0094b70be2cd6481b770248" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:35.432849 19133 leader_election.cc:343] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:35.433161 19321 tablet_bootstrap.cc:654] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:35.408859 18842 leader_election.cc:343] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:35.438917 18902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26e5db463e94d919640a07a2f8574a8" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:35.440263 20182 consensus_queue.cc:1035] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:35.453284 19131 leader_election.cc:304] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:35.454411 18713 raft_consensus.cc:1270] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Refusing update from remote peer 8784b9603f8544b79373158aace063af: Log matching property violated. Preceding OpId in replica: term: 5 index: 274. Preceding OpId from leader: term: 8 index: 280. (index mismatch)
I20250111 02:57:35.454553 20151 raft_consensus.cc:2798] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:35.455137 20151 raft_consensus.cc:491] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:35.455590 20151 raft_consensus.cc:3054] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:35.456205 20164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 274, Time since last communication: 0.000s
I20250111 02:57:35.457813 19321 tablet_bootstrap.cc:492] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:35.458204 19321 ts_tablet_manager.cc:1397] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.072s user 0.010s sys 0.003s
I20250111 02:57:35.465524 19321 raft_consensus.cc:357] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.466221 19321 raft_consensus.cc:383] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:35.466503 19321 raft_consensus.cc:738] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:35.475414 19321 consensus_queue.cc:260] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.477106 19321 ts_tablet_manager.cc:1428] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250111 02:57:35.491382 19322 ts_tablet_manager.cc:1428] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.178s user 0.007s sys 0.000s
I20250111 02:57:35.492434 19322 tablet_bootstrap.cc:492] T 2f11541b436e42b7b0728da2aaec48f7 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:35.496649 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26e5db463e94d919640a07a2f8574a8" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:35.497695 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7145efe5c0094b70be2cd6481b770248" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:35.499167 19193 raft_consensus.cc:3054] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:35.514043 19321 tablet_bootstrap.cc:492] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
W20250111 02:57:35.514554 18615 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6a03880a8b0e46b7858ed1c2c2dddcaa consensus state has no leader
I20250111 02:57:35.517625 20177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8784b9603f8544b79373158aace063af. Latest consensus state: current_term: 8 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:35.521756 19321 tablet_bootstrap.cc:654] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:35.518716 20177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: FOLLOWER
W20250111 02:57:35.524070 18989 leader_election.cc:343] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:35.503149 20046 leader_election.cc:290] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:35.538080 20188 mvcc.cc:204] Tried to move back new op lower bound from 7112967184349040640 to 7112967177802350592. Current Snapshot: MvccSnapshot[applied={T|T < 7112967184349040640}]
I20250111 02:57:35.545303 20187 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8784b9603f8544b79373158aace063af. Latest consensus state: current_term: 8 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:35.546346 20187 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:35.546856 20182 consensus_queue.cc:1035] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:35.551440 20191 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:35.552443 20191 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:35.559280 18902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "061e6eb5775443679e9cbd6b11f66fe3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:35.562275 18989 leader_election.cc:343] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:35.568717 20185 log.cc:926] Time spent T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
W20250111 02:57:35.569905 19131 leader_election.cc:343] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:35.570518 19131 leader_election.cc:304] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:35.571854 20046 consensus_queue.cc:1035] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:35.590205 20191 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:35.591405 20191 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:35.597958 19322 tablet_bootstrap.cc:654] T 2f11541b436e42b7b0728da2aaec48f7 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:35.601204 18622 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe] alter complete (version 2)
I20250111 02:57:35.604442 19193 raft_consensus.cc:2463] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:35.585961 20131 raft_consensus.cc:2743] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:35.627483 19321 tablet_bootstrap.cc:492] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:35.627920 19321 ts_tablet_manager.cc:1397] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.114s user 0.006s sys 0.015s
I20250111 02:57:35.628844 20059 raft_consensus.cc:491] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:35.629464 20059 raft_consensus.cc:513] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.631647 19321 raft_consensus.cc:357] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.632323 19321 raft_consensus.cc:383] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:35.632597 19321 raft_consensus.cc:738] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:35.633152 19321 consensus_queue.cc:260] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.632014 20059 leader_election.cc:290] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:35.636085 20059 raft_consensus.cc:491] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:35.640381 20059 raft_consensus.cc:513] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.643489 19321 ts_tablet_manager.cc:1428] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20250111 02:57:35.634526 20151 raft_consensus.cc:513] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.646040 20151 leader_election.cc:290] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:35.644487 19321 tablet_bootstrap.cc:492] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:35.655915 20059 leader_election.cc:290] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:35.655153 20151 raft_consensus.cc:491] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:35.665645 18986 leader_election.cc:304] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:35.664853 20151 raft_consensus.cc:513] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.674769 20149 raft_consensus.cc:2798] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:35.676281 20149 raft_consensus.cc:695] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
W20250111 02:57:35.704942 18615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.124:34593, user_credentials={real_user=slave}} blocked reactor thread for 110998us
I20250111 02:57:35.707397 19330 tablet_bootstrap.cc:654] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:35.677923 19321 tablet_bootstrap.cc:654] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:35.715222 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 49682.9us
W20250111 02:57:35.684350 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 91154.5us
I20250111 02:57:35.691983 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "061e6eb5775443679e9cbd6b11f66fe3" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:35.720523 19192 raft_consensus.cc:3054] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:35.712474 20149 consensus_queue.cc:237] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.738567 19321 tablet_bootstrap.cc:492] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:35.739020 19321 ts_tablet_manager.cc:1397] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.095s user 0.008s sys 0.012s
I20250111 02:57:35.742267 19321 raft_consensus.cc:357] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.742975 19321 raft_consensus.cc:383] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:35.743261 19321 raft_consensus.cc:738] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:35.743902 19321 consensus_queue.cc:260] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.746493 19321 ts_tablet_manager.cc:1428] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250111 02:57:35.746908 20203 raft_consensus.cc:491] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:35.747565 19321 tablet_bootstrap.cc:492] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:35.748208 20151 leader_election.cc:290] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:35.747588 20203 raft_consensus.cc:513] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.752146 20203 leader_election.cc:290] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:35.763636 19923 consensus_queue.cc:1035] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250111 02:57:35.765898 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 49201us
I20250111 02:57:35.768606 19321 tablet_bootstrap.cc:654] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:35.777029 19330 tablet_bootstrap.cc:492] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:35.777438 19330 ts_tablet_manager.cc:1397] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.353s user 0.004s sys 0.008s
I20250111 02:57:35.780393 19330 raft_consensus.cc:357] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.781332 19330 raft_consensus.cc:383] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:35.781590 19330 raft_consensus.cc:738] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:35.782231 19330 consensus_queue.cc:260] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.784757 19330 ts_tablet_manager.cc:1428] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250111 02:57:35.785701 19330 tablet_bootstrap.cc:492] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:35.788090 19321 tablet_bootstrap.cc:492] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:35.788516 19321 ts_tablet_manager.cc:1397] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.013s
I20250111 02:57:35.789845 20204 raft_consensus.cc:491] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:35.790390 20204 raft_consensus.cc:513] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.791118 19321 raft_consensus.cc:357] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.791777 19321 raft_consensus.cc:383] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:35.792018 19321 raft_consensus.cc:738] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:35.792616 19321 consensus_queue.cc:260] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.794684 19321 ts_tablet_manager.cc:1428] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250111 02:57:35.795584 19321 tablet_bootstrap.cc:492] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:35.812902 19321 tablet_bootstrap.cc:654] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:35.828330 20201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:35.829311 20201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:35.854269 20177 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:35.854822 20177 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:35.857018 20177 leader_election.cc:290] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8784b9603f8544b79373158aace063af (127.18.20.125:40359), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:35.859555 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 9 candidate_status { last_received { term: 8 index: 280 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:35.859643 18639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 9 candidate_status { last_received { term: 8 index: 280 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:35.863284 19595 leader_election.cc:304] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727; no voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:35.864702 20177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250111 02:57:35.866437 20204 leader_election.cc:290] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:35.896351 19192 raft_consensus.cc:2463] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:35.920961 19321 tablet_bootstrap.cc:492] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:35.921411 19321 ts_tablet_manager.cc:1397] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.126s user 0.017s sys 0.038s
I20250111 02:57:35.923769 19321 raft_consensus.cc:357] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.924480 19321 raft_consensus.cc:383] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:35.924734 19321 raft_consensus.cc:738] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:35.925334 19321 consensus_queue.cc:260] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.936858 19321 ts_tablet_manager.cc:1428] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20250111 02:57:35.937845 19321 tablet_bootstrap.cc:492] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:35.943161 19321 tablet_bootstrap.cc:654] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:35.944049 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42230a0bc94249fda91cf79fba48f024" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:35.946391 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934c900baf1b4000af7d7ec60cbb3591" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:35.952262 20195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:35.950861 18626 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: default.Table_bccba04b48cc408d845624ee7ed98e5f [id=1496848ec9594cad889312978436d9c4] alter complete (version 1)
I20250111 02:57:35.953853 19321 tablet_bootstrap.cc:492] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:35.960279 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933828fd6fb041e9a4889f4ea8a3c5ff" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:35.961900 20151 raft_consensus.cc:491] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:35.962917 20151 raft_consensus.cc:513] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.968428 19321 ts_tablet_manager.cc:1397] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.006s
I20250111 02:57:35.970907 19321 raft_consensus.cc:357] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.971630 19321 raft_consensus.cc:383] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:35.971877 19321 raft_consensus.cc:738] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:35.972260 20203 raft_consensus.cc:491] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:35.972394 19321 consensus_queue.cc:260] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.973225 20203 raft_consensus.cc:513] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:35.974468 19321 ts_tablet_manager.cc:1428] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250111 02:57:35.975700 20203 leader_election.cc:290] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:35.953272 20195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:35.967227 19050 raft_consensus.cc:3054] T 933828fd6fb041e9a4889f4ea8a3c5ff P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:35.980077 19034 tablet_service.cc:1514] Processing DeleteTablet for tablet 32afc2f158254811836fcf00f7f409ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:19 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:35.980556 19330 tablet_bootstrap.cc:654] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:35.988075 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:35.989190 19173 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf1ce65ccf54358b95665b9764fbdb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:35.989473 19165 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cd364f9a6894d19826deb59a8d8a612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:18 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:35.992105 18986 leader_election.cc:304] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:35.998358 19171 tablet_service.cc:1514] Processing DeleteTablet for tablet 32afc2f158254811836fcf00f7f409ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:19 UTC) from {username='slave'} at 127.0.0.1:32868
W20250111 02:57:35.999408 19131 leader_election.cc:343] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:35.981396 19030 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b302fcba27244c193586bc25fc9305f with delete_type TABLET_DATA_DELETED (replaced by tablet 13f0a0e7698e49a5a57562dc8ad52cd5) from {username='slave'} at 127.0.0.1:36190
W20250111 02:57:36.003409 19131 leader_election.cc:343] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.000991 19957 raft_consensus.cc:2798] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:36.005730 19957 raft_consensus.cc:695] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:36.005784 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet 74592f80afab4513a9eb2eac351c4de5 with delete_type TABLET_DATA_DELETED (replaced by tablet f57996ef57164187b7afbff68c1b934c) from {username='slave'} at 127.0.0.1:36190
W20250111 02:57:36.006762 18626 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.66:37049 (request call id 41) took 6871 ms (client timeout 1000 ms). Trace:
I20250111 02:57:35.988991 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1870cceafe463b8d024e8b065850e0" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:36.007453 19957 consensus_queue.cc:237] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.021070 20204 raft_consensus.cc:491] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:36.021732 20204 raft_consensus.cc:513] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.024858 20204 leader_election.cc:290] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:36.027762 19321 tablet_bootstrap.cc:492] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:36.000679 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb87813c28742edad99470c84269ef4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:36.030007 19171 tablet_service.cc:1514] Processing DeleteTablet for tablet 92aa64b2de5746d1b12cc93a10c9d56e with delete_type TABLET_DATA_DELETED (replaced by tablet e46894710a72407993b36801f3b90eb0) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:36.030768 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2377893bfc426fa00d5ddacd9c5961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:21 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:36.031647 19165 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:16 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:36.033973 19321 tablet_bootstrap.cc:654] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:36.040134 18903 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.006s sys 0.002s
I20250111 02:57:36.043278 18897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d837bd001aa649e9b3ca94f7cdd9f0fe" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:36.050727 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42230a0bc94249fda91cf79fba48f024" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:36.060706 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 0267a92995ea460786dc26813b5895cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:22 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:36.061756 19171 tablet_service.cc:1514] Processing DeleteTablet for tablet 62c6b166a29943ceb15f093fac728c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:27 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:36.062583 19165 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b302fcba27244c193586bc25fc9305f with delete_type TABLET_DATA_DELETED (replaced by tablet 13f0a0e7698e49a5a57562dc8ad52cd5) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:36.069204 19321 tablet_bootstrap.cc:492] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:36.069672 19321 ts_tablet_manager.cc:1397] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.001s
I20250111 02:57:36.071933 19321 raft_consensus.cc:357] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.072614 19321 raft_consensus.cc:383] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:36.072860 19321 raft_consensus.cc:738] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:36.073447 19321 consensus_queue.cc:260] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:36.009632 18626 rpcz_store.cc:269] 0111 02:57:29.135035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:29.207562 (+ 72527us) service_pool.cc:225] Handling call
0111 02:57:36.006681 (+6799119us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:34.126314 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:34.127045 (+ 731us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:34.127110 (+ 65us) write_op.cc:435] Acquired schema lock
0111 02:57:34.127128 (+ 18us) tablet.cc:659] Decoding operations
0111 02:57:34.127642 (+ 514us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:34.127706 (+ 64us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:34.127738 (+ 32us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:34.127941 (+ 203us) tablet.cc:698] Row locks acquired
0111 02:57:34.127970 (+ 29us) write_op.cc:260] PREPARE: finished
0111 02:57:34.128147 (+ 177us) write_op.cc:270] Start()
0111 02:57:34.128266 (+ 119us) write_op.cc:276] Timestamp: P: 1736564254128095 usec, L: 0
0111 02:57:34.128283 (+ 17us) op_driver.cc:348] REPLICATION: starting
0111 02:57:34.129118 (+ 835us) log.cc:844] Serialized 530 byte log entry
0111 02:57:35.556707 (+1427589us) op_driver.cc:464] REPLICATION: finished
0111 02:57:35.567902 (+ 11195us) write_op.cc:301] APPLY: starting
0111 02:57:35.567991 (+ 89us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:35.568107 (+ 116us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:35.568268 (+ 161us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:35.568492 (+ 224us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:35.568703 (+ 211us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:35.568758 (+ 55us) write_op.cc:312] APPLY: finished
0111 02:57:35.568981 (+ 223us) log.cc:844] Serialized 23 byte log entry
0111 02:57:35.571266 (+ 2285us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:35.571336 (+ 70us) write_op.cc:454] Released schema lock
0111 02:57:35.571374 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:35.577822 (+ 6448us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:35.577871 (+ 49us) write_op.cc:454] Released schema lock
Related trace 'op':
0111 02:57:35.604621 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:35.605389 (+ 768us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:35.605442 (+ 53us) write_op.cc:435] Acquired schema lock
0111 02:57:35.605477 (+ 35us) tablet.cc:659] Decoding operations
0111 02:57:35.606040 (+ 563us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:35.606158 (+ 118us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:35.606207 (+ 49us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:35.606502 (+ 295us) tablet.cc:698] Row locks acquired
0111 02:57:35.606535 (+ 33us) write_op.cc:260] PREPARE: finished
0111 02:57:35.606709 (+ 174us) write_op.cc:270] Start()
0111 02:57:35.606815 (+ 106us) write_op.cc:276] Timestamp: P: 1736564255606675 usec, L: 0
0111 02:57:35.606841 (+ 26us) op_driver.cc:348] REPLICATION: starting
0111 02:57:35.803390 (+196549us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b5400021fa0. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f316d1 00007fd427f42020 00007fd427fb2bd2 00007fd427fa82d7 00007fd427fa8114 00007fd4284143b1 00007fd428413b88 00007fd428418821 00007fd4284187d9 00007fd428418769 00007fd428418731
0111 02:57:35.804055 (+ 665us) log.cc:844] Serialized 356 byte log entry
0111 02:57:35.948590 (+144535us) op_driver.cc:464] REPLICATION: finished
0111 02:57:35.948848 (+ 258us) write_op.cc:301] APPLY: starting
0111 02:57:35.948907 (+ 59us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:35.949001 (+ 94us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:35.949018 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:35.949218 (+ 200us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:35.949361 (+ 143us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:35.949405 (+ 44us) write_op.cc:312] APPLY: finished
0111 02:57:35.949589 (+ 184us) log.cc:844] Serialized 23 byte log entry
0111 02:57:35.949741 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:35.949809 (+ 68us) write_op.cc:454] Released schema lock
0111 02:57:35.949841 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":2457,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":10976,"apply.run_cpu_time_us":1766,"apply.run_wall_time_us":9836,"num_ops":1,"prepare.queue_time_us":741,"prepare.run_cpu_time_us":3106,"prepare.run_wall_time_us":4655,"replication_time_us":1428266,"thread_start_us":17046,"threads_started":1}],["op",{"apply.queue_time_us":103,"apply.run_cpu_time_us":1175,"apply.run_wall_time_us":1175,"num_ops":1,"prepare.queue_time_us":30017,"prepare.run_cpu_time_us":7300,"prepare.run_wall_time_us":199855,"raft.queue_time_us":9545,"raft.run_cpu_time_us":717,"raft.run_wall_time_us":718,"replication_time_us":341639,"spinlock_wait_cycles":448447232,"thread_start_us":48574,"threads_started":1}]]}
I20250111 02:57:36.081526 19321 ts_tablet_manager.cc:1428] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.011s user 0.006s sys 0.002s
I20250111 02:57:36.083662 20182 consensus_queue.cc:1035] T 2e06e867266948df8eb35a85c58f578f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:36.085738 20151 leader_election.cc:290] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:36.088143 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet 199ca8692f39445f93d8c27e2134a32c with delete_type TABLET_DATA_DELETED (replaced by tablet 29f27ed2c53d43d7a51ee2e45afaf9fc) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:36.054252 20149 consensus_queue.cc:1035] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.037s
W20250111 02:57:36.091511 20211 log.cc:926] Time spent T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
I20250111 02:57:36.095400 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d837bd001aa649e9b3ca94f7cdd9f0fe" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:36.098793 19050 raft_consensus.cc:2463] T 933828fd6fb041e9a4889f4ea8a3c5ff P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:36.101033 19957 raft_consensus.cc:491] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:36.101513 19957 raft_consensus.cc:513] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.103513 19957 leader_election.cc:290] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:36.107875 19330 tablet_bootstrap.cc:492] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:36.108417 19330 ts_tablet_manager.cc:1397] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.323s user 0.012s sys 0.010s
I20250111 02:57:36.110661 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933828fd6fb041e9a4889f4ea8a3c5ff" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:36.058984 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934c900baf1b4000af7d7ec60cbb3591" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:36.117764 19330 raft_consensus.cc:357] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.118503 19330 raft_consensus.cc:383] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:36.118748 19330 raft_consensus.cc:738] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:36.119366 19330 consensus_queue.cc:260] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.121623 19330 ts_tablet_manager.cc:1428] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20250111 02:57:36.122560 19330 tablet_bootstrap.cc:492] T cb96635f30b34de48870daf5166813bc P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:36.124163 18909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2009d6f54e2842e8be3af2cc9147dc2c" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:36.125149 20131 raft_consensus.cc:491] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:36.125610 20131 raft_consensus.cc:513] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.126112 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:36.107465 18903 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s user 0.007s sys 0.002s
I20250111 02:57:36.127764 20131 leader_election.cc:290] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:36.129918 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca35147034446b3882b5244923bddc7" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:36.131454 19330 tablet_bootstrap.cc:654] T cb96635f30b34de48870daf5166813bc P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:36.137763 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb87813c28742edad99470c84269ef4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:36.145979 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1870cceafe463b8d024e8b065850e0" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:36.146952 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 81885.8us
W20250111 02:57:36.148785 19133 leader_election.cc:343] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.151288 19133 leader_election.cc:304] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.152572 20203 raft_consensus.cc:2743] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:36.153970 19133 leader_election.cc:343] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:36.161955 19133 leader_election.cc:343] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:36.162770 19133 leader_election.cc:343] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:36.169312 19133 leader_election.cc:343] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.171300 19133 leader_election.cc:304] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.173146 20204 raft_consensus.cc:2743] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:36.174297 19133 leader_election.cc:343] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.189771 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f11541b436e42b7b0728da2aaec48f7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:36.176345 19321 tablet_bootstrap.cc:492] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:36.202255 19322 tablet_bootstrap.cc:492] T 2f11541b436e42b7b0728da2aaec48f7 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:36.202968 19322 ts_tablet_manager.cc:1397] T 2f11541b436e42b7b0728da2aaec48f7 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.711s user 0.013s sys 0.009s
I20250111 02:57:36.205929 19322 raft_consensus.cc:357] T 2f11541b436e42b7b0728da2aaec48f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.208614 19321 tablet_bootstrap.cc:654] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:36.211699 20232 raft_consensus.cc:491] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:36.212261 20232 raft_consensus.cc:513] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:36.214973 18989 leader_election.cc:343] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.228943 19321 tablet_bootstrap.cc:492] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:36.229477 19321 ts_tablet_manager.cc:1397] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.053s user 0.014s sys 0.001s
I20250111 02:57:36.230230 20177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:36.232477 19321 raft_consensus.cc:357] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.233328 19321 raft_consensus.cc:383] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:36.233606 19321 raft_consensus.cc:738] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:36.230986 20177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:36.234279 19321 consensus_queue.cc:260] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.237864 19321 ts_tablet_manager.cc:1428] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250111 02:57:36.238940 19321 tablet_bootstrap.cc:492] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:36.265714 19321 tablet_bootstrap.cc:654] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:36.267899 19330 tablet_bootstrap.cc:492] T cb96635f30b34de48870daf5166813bc P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:36.268342 19330 ts_tablet_manager.cc:1397] T cb96635f30b34de48870daf5166813bc P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.146s user 0.039s sys 0.053s
I20250111 02:57:36.271391 19330 raft_consensus.cc:357] T cb96635f30b34de48870daf5166813bc P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.272089 19330 raft_consensus.cc:383] T cb96635f30b34de48870daf5166813bc P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:36.272334 19330 raft_consensus.cc:738] T cb96635f30b34de48870daf5166813bc P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:36.259120 19322 raft_consensus.cc:383] T 2f11541b436e42b7b0728da2aaec48f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:36.273240 19322 raft_consensus.cc:738] T 2f11541b436e42b7b0728da2aaec48f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:36.278184 19321 tablet_bootstrap.cc:492] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:36.278789 19321 ts_tablet_manager.cc:1397] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.000s
I20250111 02:57:36.272923 19330 consensus_queue.cc:260] T cb96635f30b34de48870daf5166813bc P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.281827 19321 raft_consensus.cc:357] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.282559 19321 raft_consensus.cc:383] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:36.282737 19330 ts_tablet_manager.cc:1428] T cb96635f30b34de48870daf5166813bc P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250111 02:57:36.282829 19321 raft_consensus.cc:738] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:36.283689 19330 tablet_bootstrap.cc:492] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:36.283450 19321 consensus_queue.cc:260] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.285607 19321 ts_tablet_manager.cc:1428] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20250111 02:57:36.213891 19133 leader_election.cc:343] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:36.211784 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 32afc2f158254811836fcf00f7f409ed because the tablet was not found. No further retry: Not found: Tablet not found: 32afc2f158254811836fcf00f7f409ed
W20250111 02:57:36.297827 18615 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 86084.9us
I20250111 02:57:36.288779 19322 consensus_queue.cc:260] T 2f11541b436e42b7b0728da2aaec48f7 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:36.298956 18615 catalog_manager.cc:4839] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): delete failed for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 because the tablet was not found. No further retry: Not found: Tablet not found: 5fe8aa967fcb4bd7a48f6f0f69bea496
W20250111 02:57:36.187922 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 5cf1ce65ccf54358b95665b9764fbdb0 because the tablet was not found. No further retry: Not found: Tablet not found: 5cf1ce65ccf54358b95665b9764fbdb0
W20250111 02:57:36.233877 18622 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.67:53171 (request call id 42) took 7121 ms (client timeout 1000 ms). Trace:
W20250111 02:57:36.300948 18614 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 113075us
W20250111 02:57:36.301787 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 2cd364f9a6894d19826deb59a8d8a612 because the tablet was not found. No further retry: Not found: Tablet not found: 2cd364f9a6894d19826deb59a8d8a612
W20250111 02:57:36.302486 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 32afc2f158254811836fcf00f7f409ed because the tablet was not found. No further retry: Not found: Tablet not found: 32afc2f158254811836fcf00f7f409ed
W20250111 02:57:36.303134 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 92aa64b2de5746d1b12cc93a10c9d56e because the tablet was not found. No further retry: Not found: Tablet not found: 92aa64b2de5746d1b12cc93a10c9d56e
W20250111 02:57:36.303805 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 1b2377893bfc426fa00d5ddacd9c5961 because the tablet was not found. No further retry: Not found: Tablet not found: 1b2377893bfc426fa00d5ddacd9c5961
W20250111 02:57:36.304414 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 5fe8aa967fcb4bd7a48f6f0f69bea496 because the tablet was not found. No further retry: Not found: Tablet not found: 5fe8aa967fcb4bd7a48f6f0f69bea496
W20250111 02:57:36.305606 18614 catalog_manager.cc:4839] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): delete failed for tablet 0267a92995ea460786dc26813b5895cf because the tablet was not found. No further retry: Not found: Tablet not found: 0267a92995ea460786dc26813b5895cf
I20250111 02:57:36.306807 19330 tablet_bootstrap.cc:654] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:36.311471 20232 leader_election.cc:290] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:36.315352 19330 tablet_bootstrap.cc:492] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:36.315793 19330 ts_tablet_manager.cc:1397] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.003s
I20250111 02:57:36.317632 19322 ts_tablet_manager.cc:1428] T 2f11541b436e42b7b0728da2aaec48f7 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.114s user 0.004s sys 0.003s
I20250111 02:57:36.318756 19322 tablet_bootstrap.cc:492] T 061e6eb5775443679e9cbd6b11f66fe3 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:36.321456 19330 raft_consensus.cc:357] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.322166 19330 raft_consensus.cc:383] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:36.322417 19330 raft_consensus.cc:738] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:36.324939 19330 consensus_queue.cc:260] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.329401 19330 ts_tablet_manager.cc:1428] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250111 02:57:36.330464 19330 tablet_bootstrap.cc:492] T 0b6687f39d4246b49c0c3eb904b11c1a P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:36.336118 19330 tablet_bootstrap.cc:654] T 0b6687f39d4246b49c0c3eb904b11c1a P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:36.337778 20240 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:36.338250 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b6687f39d4246b49c0c3eb904b11c1a" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:36.338366 20240 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:36.341681 19682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
W20250111 02:57:36.346534 18842 leader_election.cc:343] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.350232 20240 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:36.351282 19322 tablet_bootstrap.cc:654] T 061e6eb5775443679e9cbd6b11f66fe3 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:36.361932 19330 tablet_bootstrap.cc:492] T 0b6687f39d4246b49c0c3eb904b11c1a P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:36.362483 19330 ts_tablet_manager.cc:1397] T 0b6687f39d4246b49c0c3eb904b11c1a P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.007s
I20250111 02:57:36.365024 19330 raft_consensus.cc:357] T 0b6687f39d4246b49c0c3eb904b11c1a P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.365980 19330 raft_consensus.cc:383] T 0b6687f39d4246b49c0c3eb904b11c1a P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:36.366320 19330 raft_consensus.cc:738] T 0b6687f39d4246b49c0c3eb904b11c1a P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:36.367699 19330 consensus_queue.cc:260] T 0b6687f39d4246b49c0c3eb904b11c1a P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.370465 19330 ts_tablet_manager.cc:1428] T 0b6687f39d4246b49c0c3eb904b11c1a P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20250111 02:57:36.371644 19330 tablet_bootstrap.cc:492] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
W20250111 02:57:36.300837 18622 rpcz_store.cc:269] 0111 02:57:29.112648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:29.130535 (+ 17887us) service_pool.cc:225] Handling call
0111 02:57:29.157319 (+ 26784us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d51dee 00007fd427aaffc4 00007fd427aaff52 00007fd427aafe81 00007fd427aafdfc 00007fd427aaf0b2 00007fd425d79dfc 00007fd425d79236
0111 02:57:36.233796 (+7076477us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:32.926936 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:32.927463 (+ 527us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:32.927502 (+ 39us) write_op.cc:435] Acquired schema lock
0111 02:57:32.927519 (+ 17us) tablet.cc:659] Decoding operations
0111 02:57:32.928165 (+ 646us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:32.928248 (+ 83us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:32.928276 (+ 28us) tablet.cc:682] Acquiring locks for 3 operations
0111 02:57:32.928598 (+ 322us) tablet.cc:698] Row locks acquired
0111 02:57:32.928615 (+ 17us) write_op.cc:260] PREPARE: finished
0111 02:57:32.928751 (+ 136us) write_op.cc:270] Start()
0111 02:57:32.928839 (+ 88us) write_op.cc:276] Timestamp: P: 1736564252928713 usec, L: 0
0111 02:57:32.928857 (+ 18us) op_driver.cc:348] REPLICATION: starting
0111 02:57:32.929432 (+ 575us) log.cc:844] Serialized 1284 byte log entry
0111 02:57:33.868219 (+938787us) op_driver.cc:464] REPLICATION: finished
0111 02:57:33.878183 (+ 9964us) write_op.cc:301] APPLY: starting
0111 02:57:33.878256 (+ 73us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:33.878380 (+ 124us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:33.878479 (+ 99us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:33.878969 (+ 490us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:33.879153 (+ 184us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0111 02:57:33.879198 (+ 45us) write_op.cc:312] APPLY: finished
0111 02:57:33.879491 (+ 293us) log.cc:844] Serialized 35 byte log entry
0111 02:57:33.933995 (+ 54504us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:33.934106 (+ 111us) write_op.cc:454] Released schema lock
0111 02:57:33.934172 (+ 66us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:33.934560 (+ 388us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:33.934592 (+ 32us) write_op.cc:454] Released schema lock
Related trace 'op':
0111 02:57:33.990119 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:33.990653 (+ 534us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:33.990683 (+ 30us) write_op.cc:435] Acquired schema lock
0111 02:57:33.990697 (+ 14us) tablet.cc:659] Decoding operations
0111 02:57:33.991023 (+ 326us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:33.991111 (+ 88us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:33.991138 (+ 27us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:33.991339 (+ 201us) tablet.cc:698] Row locks acquired
0111 02:57:33.991355 (+ 16us) write_op.cc:260] PREPARE: finished
0111 02:57:33.991513 (+ 158us) write_op.cc:270] Start()
0111 02:57:33.991634 (+ 121us) write_op.cc:276] Timestamp: P: 1736564253991465 usec, L: 0
0111 02:57:33.991651 (+ 17us) op_driver.cc:348] REPLICATION: starting
0111 02:57:33.992342 (+ 691us) log.cc:844] Serialized 356 byte log entry
0111 02:57:34.257068 (+264726us) op_driver.cc:464] REPLICATION: finished
0111 02:57:34.257659 (+ 591us) write_op.cc:301] APPLY: starting
0111 02:57:34.257735 (+ 76us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:34.257883 (+ 148us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:34.258000 (+ 117us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:34.258287 (+ 287us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:34.258475 (+ 188us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:34.258538 (+ 63us) write_op.cc:312] APPLY: finished
0111 02:57:34.258786 (+ 248us) log.cc:844] Serialized 23 byte log entry
0111 02:57:34.258987 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:34.259080 (+ 93us) write_op.cc:454] Released schema lock
0111 02:57:34.259145 (+ 65us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:34.259454 (+ 309us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:34.259491 (+ 37us) write_op.cc:454] Released schema lock
Related trace 'op':
0111 02:57:34.268293 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:34.268959 (+ 666us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:34.268999 (+ 40us) write_op.cc:435] Acquired schema lock
0111 02:57:34.269028 (+ 29us) tablet.cc:659] Decoding operations
0111 02:57:34.269636 (+ 608us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:34.269690 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:34.269740 (+ 50us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:34.269963 (+ 223us) tablet.cc:698] Row locks acquired
0111 02:57:34.269990 (+ 27us) write_op.cc:260] PREPARE: finished
0111 02:57:34.270171 (+ 181us) write_op.cc:270] Start()
0111 02:57:34.270307 (+ 136us) write_op.cc:276] Timestamp: P: 1736564254270128 usec, L: 0
0111 02:57:34.270342 (+ 35us) op_driver.cc:348] REPLICATION: starting
0111 02:57:34.271155 (+ 813us) log.cc:844] Serialized 356 byte log entry
0111 02:57:35.574251 (+1303096us) op_driver.cc:464] REPLICATION: finished
0111 02:57:35.578238 (+ 3987us) write_op.cc:301] APPLY: starting
0111 02:57:35.578317 (+ 79us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:35.578480 (+ 163us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:35.578638 (+ 158us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:35.578957 (+ 319us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:35.598349 (+ 19392us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:35.598416 (+ 67us) write_op.cc:312] APPLY: finished
0111 02:57:35.598691 (+ 275us) log.cc:844] Serialized 23 byte log entry
0111 02:57:35.600671 (+ 1980us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:35.600753 (+ 82us) write_op.cc:454] Released schema lock
0111 02:57:35.600806 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:35.601105 (+ 299us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:35.601138 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":65,"reported_tablets":26,"spinlock_wait_cycles":50329856,"child_traces":[["op",{"apply.queue_time_us":9714,"apply.run_cpu_time_us":2068,"apply.run_wall_time_us":56302,"num_ops":3,"prepare.queue_time_us":274396,"prepare.run_cpu_time_us":2794,"prepare.run_wall_time_us":4747,"raft.queue_time_us":3679,"raft.run_cpu_time_us":645,"raft.run_wall_time_us":645,"replication_time_us":939229,"thread_start_us":288731,"threads_started":2}],["op",{"apply.queue_time_us":353,"apply.run_cpu_time_us":1742,"apply.run_wall_time_us":1741,"num_ops":1,"prepare.queue_time_us":52499,"prepare.run_cpu_time_us":2486,"prepare.run_wall_time_us":2517,"replication_time_us":265294}],["op",{"apply.queue_time_us":3779,"apply.run_cpu_time_us":1958,"apply.run_wall_time_us":22823,"num_ops":1,"prepare.queue_time_us":306,"prepare.run_cpu_time_us":3085,"prepare.run_wall_time_us":4802,"replication_time_us":1303790,"thread_start_us":13890,"threads_started":1}]]}
W20250111 02:57:36.385880 18614 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 80284.9us
I20250111 02:57:36.387106 20070 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
W20250111 02:57:36.387878 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 389084us
I20250111 02:57:36.389074 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca35147034446b3882b5244923bddc7" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:36.391783 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2009d6f54e2842e8be3af2cc9147dc2c" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:36.295284 19133 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 87299.4us
W20250111 02:57:36.394528 19133 leader_election.cc:343] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.395663 19330 tablet_bootstrap.cc:654] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:36.397323 19322 tablet_bootstrap.cc:492] T 061e6eb5775443679e9cbd6b11f66fe3 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:36.397742 19322 ts_tablet_manager.cc:1397] T 061e6eb5775443679e9cbd6b11f66fe3 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.008s
I20250111 02:57:36.403703 20233 consensus_queue.cc:1035] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:36.409811 19330 tablet_bootstrap.cc:492] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:36.410344 19330 ts_tablet_manager.cc:1397] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.002s
I20250111 02:57:36.413766 19330 raft_consensus.cc:357] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.414575 19330 raft_consensus.cc:383] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:36.414887 19330 raft_consensus.cc:738] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:36.396113 20221 tablet_replica.cc:331] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:36.415532 19330 consensus_queue.cc:260] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.417361 20221 raft_consensus.cc:2238] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:36.178494 20222 tablet_replica.cc:331] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:36.418869 19330 ts_tablet_manager.cc:1428] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250111 02:57:36.422513 20222 raft_consensus.cc:2238] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:36.435564 20222 raft_consensus.cc:2267] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:36.437688 20222 ts_tablet_manager.cc:1905] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:36.428560 19321 tablet_bootstrap.cc:492] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
W20250111 02:57:36.422945 19133 leader_election.cc:343] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.441493 20204 raft_consensus.cc:491] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:36.441506 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f71f8dd2894d40b3bfdd20c1453787" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:36.442021 20204 raft_consensus.cc:513] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.445005 19321 tablet_bootstrap.cc:654] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:36.444947 20204 leader_election.cc:290] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:36.454679 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b6687f39d4246b49c0c3eb904b11c1a" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:36.455494 19194 raft_consensus.cc:2463] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:36.457587 19322 raft_consensus.cc:357] T 061e6eb5775443679e9cbd6b11f66fe3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.458355 19322 raft_consensus.cc:383] T 061e6eb5775443679e9cbd6b11f66fe3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:36.458619 19322 raft_consensus.cc:738] T 061e6eb5775443679e9cbd6b11f66fe3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:36.461766 18841 leader_election.cc:304] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f
I20250111 02:57:36.462582 20233 raft_consensus.cc:2798] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:36.462939 20233 raft_consensus.cc:491] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:36.463276 20233 raft_consensus.cc:3054] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:36.463915 20232 consensus_queue.cc:1035] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250111 02:57:36.464344 19133 leader_election.cc:343] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.465824 20221 raft_consensus.cc:2267] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:36.466276 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f71f8dd2894d40b3bfdd20c1453787" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:36.462141 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873325cece64db6a673d4fe08eb7409" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:36.469235 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873325cece64db6a673d4fe08eb7409" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:36.468901 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:36.472159 19050 raft_consensus.cc:2463] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:36.483371 19322 consensus_queue.cc:260] T 061e6eb5775443679e9cbd6b11f66fe3 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.486953 19322 ts_tablet_manager.cc:1428] T 061e6eb5775443679e9cbd6b11f66fe3 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.089s user 0.007s sys 0.000s
W20250111 02:57:36.487602 19133 leader_election.cc:343] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.489526 19330 tablet_bootstrap.cc:492] T b4273afc264843029ed0baf1fc5c8a3e P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:36.497005 20233 raft_consensus.cc:513] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.469625 20221 ts_tablet_manager.cc:1905] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 02:57:36.521744 19131 leader_election.cc:343] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.522152 19131 leader_election.cc:304] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.535770 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f11541b436e42b7b0728da2aaec48f7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:36.537117 19193 raft_consensus.cc:2463] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
W20250111 02:57:36.539093 19131 leader_election.cc:343] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.540164 19131 leader_election.cc:304] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.539673 20151 raft_consensus.cc:2743] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:36.541152 20151 raft_consensus.cc:2743] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:36.542281 19131 leader_election.cc:343] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.542754 19131 leader_election.cc:304] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.520164 20235 raft_consensus.cc:491] T 061e6eb5775443679e9cbd6b11f66fe3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:36.543581 20151 raft_consensus.cc:2743] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:36.543795 20235 raft_consensus.cc:513] T 061e6eb5775443679e9cbd6b11f66fe3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.544622 19131 leader_election.cc:304] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.546592 20235 leader_election.cc:290] T 061e6eb5775443679e9cbd6b11f66fe3 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:36.547516 20151 raft_consensus.cc:2798] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:36.548043 20151 raft_consensus.cc:695] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:36.548905 20151 consensus_queue.cc:237] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.550690 19321 tablet_bootstrap.cc:492] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
W20250111 02:57:36.556182 18615 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6a03880a8b0e46b7858ed1c2c2dddcaa consensus state has no leader
I20250111 02:57:36.566341 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "061e6eb5775443679e9cbd6b11f66fe3" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:36.570973 19131 leader_election.cc:343] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.571565 19131 leader_election.cc:304] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.572291 20204 raft_consensus.cc:2743] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:36.573340 20233 leader_election.cc:290] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:36.573784 19321 ts_tablet_manager.cc:1397] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.145s user 0.013s sys 0.003s
W20250111 02:57:36.585171 19131 leader_election.cc:343] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.585788 19131 leader_election.cc:304] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
W20250111 02:57:36.586993 18989 consensus_peers.cc:487] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:36.593382 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b6687f39d4246b49c0c3eb904b11c1a" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:36.593328 20235 consensus_queue.cc:1035] T e4bd42ad66064bcaa043746081f1c759 P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:36.594321 19050 raft_consensus.cc:3054] T 0b6687f39d4246b49c0c3eb904b11c1a P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:36.589228 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "061e6eb5775443679e9cbd6b11f66fe3" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:36.599646 19193 raft_consensus.cc:2388] T 061e6eb5775443679e9cbd6b11f66fe3 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:36.589515 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b6687f39d4246b49c0c3eb904b11c1a" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:36.600518 19192 raft_consensus.cc:3054] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:36.604245 19054 raft_consensus.cc:3054] T cb96635f30b34de48870daf5166813bc P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:36.605129 18841 leader_election.cc:304] T 061e6eb5775443679e9cbd6b11f66fe3 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:36.607607 20151 raft_consensus.cc:2743] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:36.609265 19192 raft_consensus.cc:2463] T 0b6687f39d4246b49c0c3eb904b11c1a P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:36.529791 19322 tablet_bootstrap.cc:492] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
W20250111 02:57:36.611665 19131 leader_election.cc:343] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.611534 18841 leader_election.cc:304] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:36.612187 20232 raft_consensus.cc:3054] T 061e6eb5775443679e9cbd6b11f66fe3 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:36.613823 19330 tablet_bootstrap.cc:654] T b4273afc264843029ed0baf1fc5c8a3e P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:36.616132 20046 consensus_queue.cc:1035] T 0ef86e1fd412484db83e7d39c9059a90 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250111 02:57:36.612140 19131 leader_election.cc:304] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.621564 19321 raft_consensus.cc:357] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.622390 19321 raft_consensus.cc:383] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:36.622675 19321 raft_consensus.cc:738] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:36.623181 20151 raft_consensus.cc:2743] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:36.635247 20252 raft_consensus.cc:2798] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:36.635973 20252 raft_consensus.cc:695] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [term 1 LEADER]: Becoming Leader. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Running, Role: LEADER
I20250111 02:57:36.637176 20252 consensus_queue.cc:237] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.643046 18986 leader_election.cc:304] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.643867 20046 raft_consensus.cc:2798] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:36.644290 20046 raft_consensus.cc:491] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:36.644755 20046 raft_consensus.cc:3054] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:36.640122 19321 consensus_queue.cc:260] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.654682 20204 raft_consensus.cc:491] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:36.655472 20204 raft_consensus.cc:513] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.642769 20151 raft_consensus.cc:491] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:36.657346 20151 raft_consensus.cc:513] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.658272 20204 leader_election.cc:290] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:36.660416 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01771c56ea5498a876d3bb865c752a2" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:36.661856 19133 leader_election.cc:343] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.662576 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64abb30da7fc452cad774bbe59b498f4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:36.662741 20151 leader_election.cc:290] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:36.664137 19133 leader_election.cc:343] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.669457 20222 ts_tablet_manager.cc:1918] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:36.669979 20222 log.cc:1198] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/62c6b166a29943ceb15f093fac728c79
I20250111 02:57:36.676265 19321 ts_tablet_manager.cc:1428] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.091s user 0.006s sys 0.001s
I20250111 02:57:36.677330 19321 tablet_bootstrap.cc:492] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:36.678952 20204 raft_consensus.cc:491] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:36.679579 20204 raft_consensus.cc:513] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.682545 20204 leader_election.cc:290] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:36.682469 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f561cfd4f34426b7e42027e53b3ff1" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:36.683894 19133 leader_election.cc:343] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.685750 20232 raft_consensus.cc:2743] T 061e6eb5775443679e9cbd6b11f66fe3 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:36.659834 20233 consensus_queue.cc:1035] T cb96635f30b34de48870daf5166813bc P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
W20250111 02:57:36.688608 18842 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 49995us
I20250111 02:57:36.698370 19322 tablet_bootstrap.cc:654] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:36.719743 20222 ts_tablet_manager.cc:1939] T 62c6b166a29943ceb15f093fac728c79 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
W20250111 02:57:36.723662 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 50712us
W20250111 02:57:36.728263 19131 leader_election.cc:343] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.728736 19131 leader_election.cc:304] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.729756 19131 leader_election.cc:304] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.731948 20151 raft_consensus.cc:2743] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:36.732528 20151 raft_consensus.cc:2798] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:36.732836 20151 raft_consensus.cc:491] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:36.733129 20151 raft_consensus.cc:3054] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:36.738399 20046 raft_consensus.cc:513] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.742215 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64abb30da7fc452cad774bbe59b498f4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:36.743191 19330 tablet_bootstrap.cc:492] T b4273afc264843029ed0baf1fc5c8a3e P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:36.740810 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01771c56ea5498a876d3bb865c752a2" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:36.743654 19330 ts_tablet_manager.cc:1397] T b4273afc264843029ed0baf1fc5c8a3e P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.254s user 0.008s sys 0.011s
I20250111 02:57:36.743927 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f561cfd4f34426b7e42027e53b3ff1" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:36.745808 20046 leader_election.cc:290] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:36.747321 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f11541b436e42b7b0728da2aaec48f7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:36.747965 18899 raft_consensus.cc:3054] T 2f11541b436e42b7b0728da2aaec48f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:36.764001 19050 raft_consensus.cc:2463] T 0b6687f39d4246b49c0c3eb904b11c1a P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:36.779017 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f11541b436e42b7b0728da2aaec48f7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:36.779704 19195 raft_consensus.cc:3054] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:36.780526 18899 raft_consensus.cc:2463] T 2f11541b436e42b7b0728da2aaec48f7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:36.780620 20204 raft_consensus.cc:491] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:36.781061 20204 raft_consensus.cc:513] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.781630 18989 leader_election.cc:304] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:36.781682 19923 consensus_queue.cc:1035] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:36.783023 19923 raft_consensus.cc:2798] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:36.783921 19923 raft_consensus.cc:695] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:36.784433 20204 leader_election.cc:290] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:36.784909 19923 consensus_queue.cc:237] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.785133 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6da575f052d4b20a8e41f0bd9dab7e4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:36.786705 19133 leader_election.cc:343] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.795048 19322 tablet_bootstrap.cc:492] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:36.795527 19322 ts_tablet_manager.cc:1397] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.266s user 0.012s sys 0.000s
W20250111 02:57:36.803385 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 73702.5us
I20250111 02:57:36.807833 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6da575f052d4b20a8e41f0bd9dab7e4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:36.822412 19131 leader_election.cc:343] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.824229 19131 leader_election.cc:304] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.824714 19322 raft_consensus.cc:357] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.826766 20204 raft_consensus.cc:2743] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:36.826954 19322 raft_consensus.cc:383] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:36.827247 19322 raft_consensus.cc:738] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:36.827937 19322 consensus_queue.cc:260] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:36.828109 19131 leader_election.cc:343] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.833077 19131 leader_election.cc:304] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.833830 19195 raft_consensus.cc:2463] T 2f11541b436e42b7b0728da2aaec48f7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:36.834551 20151 raft_consensus.cc:513] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.835703 20046 raft_consensus.cc:491] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:36.836162 20046 raft_consensus.cc:513] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:36.836719 18989 consensus_peers.cc:487] T b26e5db463e94d919640a07a2f8574a8 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:36.837080 20221 ts_tablet_manager.cc:1918] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:36.837515 20221 log.cc:1198] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/4b302fcba27244c193586bc25fc9305f
I20250111 02:57:36.837790 20151 leader_election.cc:290] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:36.838160 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:36.838135 20046 leader_election.cc:290] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:36.839254 20054 consensus_queue.cc:1035] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250111 02:57:36.839617 19133 leader_election.cc:343] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.840983 20221 ts_tablet_manager.cc:1939] T 4b302fcba27244c193586bc25fc9305f P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:36.842327 20204 raft_consensus.cc:2743] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:36.843559 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d837bd001aa649e9b3ca94f7cdd9f0fe" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:36.846020 19321 tablet_bootstrap.cc:654] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:36.843472 19131 leader_election.cc:343] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.846951 19131 leader_election.cc:304] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.851451 20204 raft_consensus.cc:2743] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:36.852592 19131 leader_election.cc:343] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.855329 19131 leader_election.cc:304] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.856230 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d837bd001aa649e9b3ca94f7cdd9f0fe" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:36.857107 19194 raft_consensus.cc:2463] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:36.859272 20204 raft_consensus.cc:2743] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:36.862565 19322 ts_tablet_manager.cc:1428] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.067s user 0.005s sys 0.000s
W20250111 02:57:36.841763 18986 consensus_peers.cc:487] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62c6b166a29943ceb15f093fac728c79. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:36.871424 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:36.872016 19050 raft_consensus.cc:3054] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:36.844700 18989 leader_election.cc:343] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.882208 19050 raft_consensus.cc:2463] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:36.793207 19330 raft_consensus.cc:357] T b4273afc264843029ed0baf1fc5c8a3e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.883436 19131 leader_election.cc:304] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.883710 19330 raft_consensus.cc:383] T b4273afc264843029ed0baf1fc5c8a3e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:36.884027 19330 raft_consensus.cc:738] T b4273afc264843029ed0baf1fc5c8a3e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:36.884361 20204 raft_consensus.cc:2798] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:36.884809 20204 raft_consensus.cc:695] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:36.884651 19330 consensus_queue.cc:260] T b4273afc264843029ed0baf1fc5c8a3e P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.885572 20204 consensus_queue.cc:237] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.887445 19330 ts_tablet_manager.cc:1428] T b4273afc264843029ed0baf1fc5c8a3e P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.144s user 0.007s sys 0.001s
I20250111 02:57:36.899959 19330 tablet_bootstrap.cc:492] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:36.923393 19322 tablet_bootstrap.cc:492] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:36.912675 20270 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:36.923936 20270 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:36.927827 20270 leader_election.cc:290] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8784b9603f8544b79373158aace063af (127.18.20.125:40359), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:36.928442 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:36.933478 19330 tablet_bootstrap.cc:654] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:36.935084 18986 leader_election.cc:304] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:36.935516 19321 tablet_bootstrap.cc:492] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:36.935948 19321 ts_tablet_manager.cc:1397] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.259s user 0.006s sys 0.008s
I20250111 02:57:36.936244 20046 raft_consensus.cc:2798] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:36.936591 20046 raft_consensus.cc:491] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:36.936898 20046 raft_consensus.cc:3054] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:36.938319 19321 raft_consensus.cc:357] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.938989 19321 raft_consensus.cc:383] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:36.947424 19321 raft_consensus.cc:738] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:36.948134 19321 consensus_queue.cc:260] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.950681 19321 ts_tablet_manager.cc:1428] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20250111 02:57:36.951682 19321 tablet_bootstrap.cc:492] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:36.950714 20046 raft_consensus.cc:513] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:36.981837 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d837bd001aa649e9b3ca94f7cdd9f0fe" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:36.991750 18989 leader_election.cc:343] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:36.996352 19330 tablet_bootstrap.cc:492] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:36.996824 19330 ts_tablet_manager.cc:1397] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.097s user 0.009s sys 0.004s
I20250111 02:57:37.002475 19321 tablet_bootstrap.cc:654] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:37.012212 19330 raft_consensus.cc:357] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.013571 19330 raft_consensus.cc:383] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:37.013854 19330 raft_consensus.cc:738] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:37.015630 19330 consensus_queue.cc:260] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.018563 19322 tablet_bootstrap.cc:654] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:37.023694 20182 consensus_queue.cc:1035] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:37.028184 20046 leader_election.cc:290] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:37.038180 19321 tablet_bootstrap.cc:492] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:37.038681 19321 ts_tablet_manager.cc:1397] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.087s user 0.006s sys 0.003s
I20250111 02:57:37.040577 20204 raft_consensus.cc:491] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:37.041558 20204 raft_consensus.cc:513] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:37.042050 18693 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.697s (SENT)
I20250111 02:57:37.046859 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d83cfd85a52f438dabe49b6223d5090f" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:37.050119 19133 leader_election.cc:343] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:37.052507 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d83cfd85a52f438dabe49b6223d5090f" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:37.053825 19131 leader_election.cc:343] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:37.054243 19131 leader_election.cc:304] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:37.056306 20151 raft_consensus.cc:2743] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:37.057483 20204 leader_election.cc:290] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:37.040833 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d837bd001aa649e9b3ca94f7cdd9f0fe" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:37.058596 19195 raft_consensus.cc:3054] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:37.061600 19330 ts_tablet_manager.cc:1428] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.064s user 0.002s sys 0.005s
I20250111 02:57:37.065531 19195 raft_consensus.cc:2463] T d837bd001aa649e9b3ca94f7cdd9f0fe P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:37.073271 18986 leader_election.cc:304] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:37.075433 20148 raft_consensus.cc:2798] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:37.075927 20148 raft_consensus.cc:695] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:37.076674 20148 consensus_queue.cc:237] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.099967 20204 raft_consensus.cc:491] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:37.100478 20204 raft_consensus.cc:513] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.102566 20204 leader_election.cc:290] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:37.104115 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa86b1e6eca7403c9a9c52592b8fb636" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:37.104553 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa86b1e6eca7403c9a9c52592b8fb636" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:37.105795 19131 leader_election.cc:343] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:37.106292 19133 leader_election.cc:343] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:37.106690 19133 leader_election.cc:304] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:37.108306 20204 raft_consensus.cc:2743] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:37.149232 19330 tablet_bootstrap.cc:492] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:37.167768 19322 tablet_bootstrap.cc:492] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:37.168224 19322 ts_tablet_manager.cc:1397] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.245s user 0.012s sys 0.002s
I20250111 02:57:37.170701 19322 raft_consensus.cc:357] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.171453 19322 raft_consensus.cc:383] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:37.171702 19322 raft_consensus.cc:738] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:37.172046 20148 consensus_queue.cc:1035] T 061e6eb5775443679e9cbd6b11f66fe3 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:37.172279 19322 consensus_queue.cc:260] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.175679 19321 raft_consensus.cc:357] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.176373 19321 raft_consensus.cc:383] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:37.176625 19321 raft_consensus.cc:738] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:37.177172 19321 consensus_queue.cc:260] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.208678 19330 tablet_bootstrap.cc:654] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:37.211186 19321 ts_tablet_manager.cc:1428] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.172s user 0.012s sys 0.000s
I20250111 02:57:37.225842 19321 tablet_bootstrap.cc:492] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:37.233562 19322 ts_tablet_manager.cc:1428] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.065s user 0.006s sys 0.000s
I20250111 02:57:37.234670 19322 tablet_bootstrap.cc:492] T adf61f6cb63d4e8ea46b440e7ecce00d P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:37.308820 19330 tablet_bootstrap.cc:492] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:37.309257 19330 ts_tablet_manager.cc:1397] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.177s user 0.014s sys 0.000s
I20250111 02:57:37.311632 19330 raft_consensus.cc:357] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.312536 19330 raft_consensus.cc:383] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:37.315905 19321 tablet_bootstrap.cc:654] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:37.312781 19330 raft_consensus.cc:738] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:37.319216 19330 consensus_queue.cc:260] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.321465 19330 ts_tablet_manager.cc:1428] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250111 02:57:37.332381 19330 tablet_bootstrap.cc:492] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:37.343636 19330 tablet_bootstrap.cc:654] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:37.361781 19322 tablet_bootstrap.cc:654] T adf61f6cb63d4e8ea46b440e7ecce00d P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:37.365928 20148 raft_consensus.cc:491] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:37.366422 20148 raft_consensus.cc:513] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.371377 19330 tablet_bootstrap.cc:492] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:37.371838 19330 ts_tablet_manager.cc:1397] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.000s
I20250111 02:57:37.376117 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934c900baf1b4000af7d7ec60cbb3591" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:37.377559 18989 leader_election.cc:343] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:37.378648 19330 raft_consensus.cc:357] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.379351 19330 raft_consensus.cc:383] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:37.379592 19330 raft_consensus.cc:738] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:37.380870 19330 consensus_queue.cc:260] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.382550 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934c900baf1b4000af7d7ec60cbb3591" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:37.383198 19195 raft_consensus.cc:2463] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:37.384387 18986 leader_election.cc:304] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:37.385195 20182 raft_consensus.cc:2798] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:37.385581 20182 raft_consensus.cc:491] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:37.385946 20182 raft_consensus.cc:3054] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:37.382802 20148 leader_election.cc:290] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:37.401651 20232 raft_consensus.cc:491] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:37.402156 20232 raft_consensus.cc:513] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.404059 20232 leader_election.cc:290] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:37.404922 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4273afc264843029ed0baf1fc5c8a3e" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:37.405521 19195 raft_consensus.cc:2463] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:37.406678 18841 leader_election.cc:304] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:37.407789 20204 raft_consensus.cc:491] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:37.408270 20204 raft_consensus.cc:513] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.410188 20204 leader_election.cc:290] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:37.410503 19330 ts_tablet_manager.cc:1428] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.038s user 0.008s sys 0.000s
I20250111 02:57:37.411546 19330 tablet_bootstrap.cc:492] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:37.412135 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff06134303d4cb0b4292fe1c0c2d443" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:37.413046 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4273afc264843029ed0baf1fc5c8a3e" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:37.413690 19050 raft_consensus.cc:2463] T b4273afc264843029ed0baf1fc5c8a3e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
W20250111 02:57:37.413698 19133 leader_election.cc:343] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:37.415688 20232 raft_consensus.cc:2798] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:37.416093 20232 raft_consensus.cc:491] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:37.416376 20232 raft_consensus.cc:3054] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:37.417568 19330 tablet_bootstrap.cc:654] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:37.427706 19133 consensus_peers.cc:487] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:37.429755 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff06134303d4cb0b4292fe1c0c2d443" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:37.454355 19131 leader_election.cc:343] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:37.454823 19131 leader_election.cc:304] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:37.456003 20204 raft_consensus.cc:2743] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:37.478344 19330 tablet_bootstrap.cc:492] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:37.478777 19330 ts_tablet_manager.cc:1397] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.002s
I20250111 02:57:37.489533 19330 raft_consensus.cc:357] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.490609 19330 raft_consensus.cc:383] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:37.490861 19330 raft_consensus.cc:738] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:37.491483 19330 consensus_queue.cc:260] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.493140 19330 ts_tablet_manager.cc:1428] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
W20250111 02:57:37.495630 19594 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.568s (SENT) [suppressed 2 similar messages]
I20250111 02:57:37.516978 19321 tablet_bootstrap.cc:492] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:37.517437 19321 ts_tablet_manager.cc:1397] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.294s user 0.019s sys 0.001s
I20250111 02:57:37.518795 20148 raft_consensus.cc:491] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:37.519338 20148 raft_consensus.cc:513] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.521325 20148 leader_election.cc:290] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:37.522948 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1870cceafe463b8d024e8b065850e0" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:37.524838 18989 leader_election.cc:343] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:37.604892 19330 tablet_bootstrap.cc:492] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:37.608894 20229 raft_consensus.cc:491] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:37.610970 20229 raft_consensus.cc:513] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.620579 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42230a0bc94249fda91cf79fba48f024" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:37.625903 18989 leader_election.cc:343] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:37.618986 20229 leader_election.cc:290] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:37.641599 19330 tablet_bootstrap.cc:654] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:37.648501 18614 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 1ef92ac716824107916a1ea4fd49d5aa (table default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1ef92ac716824107916a1ea4fd49d5aa consensus state has no leader
I20250111 02:57:37.688226 19321 raft_consensus.cc:357] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.695997 19322 tablet_bootstrap.cc:492] T adf61f6cb63d4e8ea46b440e7ecce00d P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:37.696511 19322 ts_tablet_manager.cc:1397] T adf61f6cb63d4e8ea46b440e7ecce00d P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.462s user 0.019s sys 0.014s
I20250111 02:57:37.635257 20232 raft_consensus.cc:513] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.715917 20254 consensus_queue.cc:1035] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:37.742434 19322 raft_consensus.cc:357] T adf61f6cb63d4e8ea46b440e7ecce00d P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.749293 19322 raft_consensus.cc:383] T adf61f6cb63d4e8ea46b440e7ecce00d P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:37.750283 19322 raft_consensus.cc:738] T adf61f6cb63d4e8ea46b440e7ecce00d P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:37.752641 19322 consensus_queue.cc:260] T adf61f6cb63d4e8ea46b440e7ecce00d P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.771627 20232 leader_election.cc:290] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:37.777766 18693 leader_election.cc:336] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.697s (SENT)
I20250111 02:57:37.779911 18693 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:37.796175 20295 raft_consensus.cc:491] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:37.798152 20295 raft_consensus.cc:513] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.805236 20295 leader_election.cc:290] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:37.807384 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4273afc264843029ed0baf1fc5c8a3e" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:37.825114 19330 tablet_bootstrap.cc:492] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:37.828104 19330 ts_tablet_manager.cc:1397] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.224s user 0.030s sys 0.006s
I20250111 02:57:37.827312 19050 raft_consensus.cc:3054] T b4273afc264843029ed0baf1fc5c8a3e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:37.780079 19322 ts_tablet_manager.cc:1428] T adf61f6cb63d4e8ea46b440e7ecce00d P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.083s user 0.018s sys 0.003s
I20250111 02:57:37.826432 18900 raft_consensus.cc:3054] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:37.828910 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7145efe5c0094b70be2cd6481b770248" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:37.845885 19322 tablet_bootstrap.cc:492] T b26e5db463e94d919640a07a2f8574a8 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
W20250111 02:57:37.852772 18989 leader_election.cc:343] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:37.697589 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1870cceafe463b8d024e8b065850e0" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:37.872264 19194 raft_consensus.cc:2463] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:37.731894 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4273afc264843029ed0baf1fc5c8a3e" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:37.875365 19192 raft_consensus.cc:3054] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:37.886607 19321 raft_consensus.cc:383] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:37.913108 19321 raft_consensus.cc:738] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:37.806465 20297 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250111 02:57:37.723392 20182 raft_consensus.cc:513] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.948390 19330 raft_consensus.cc:357] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:37.949565 19330 raft_consensus.cc:383] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:37.949725 20182 leader_election.cc:290] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:37.949914 19330 raft_consensus.cc:738] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
W20250111 02:57:37.950522 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 97939.6us
I20250111 02:57:37.955006 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42230a0bc94249fda91cf79fba48f024" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:37.956245 19194 raft_consensus.cc:2463] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:37.957700 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7145efe5c0094b70be2cd6481b770248" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:37.958575 19193 raft_consensus.cc:2463] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:38.028225 19322 tablet_bootstrap.cc:654] T b26e5db463e94d919640a07a2f8574a8 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:37.916154 19321 consensus_queue.cc:260] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.039731 20204 raft_consensus.cc:491] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250111 02:57:38.049185 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 46114.3us
I20250111 02:57:37.950747 19330 consensus_queue.cc:260] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.058189 20204 raft_consensus.cc:513] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.066661 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c669a3ff8aa46fa8b04b86ed3bfde49" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:38.066975 19321 ts_tablet_manager.cc:1428] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.549s user 0.018s sys 0.000s
W20250111 02:57:38.068711 19321 ts_tablet_manager.cc:1460] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa: Tablet startup took 550ms
W20250111 02:57:38.072425 19133 leader_election.cc:343] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:38.072172 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934c900baf1b4000af7d7ec60cbb3591" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:38.084671 19594 leader_election.cc:336] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.568s (SENT)
I20250111 02:57:38.089174 19594 leader_election.cc:304] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727; no voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:38.038798 19050 raft_consensus.cc:2463] T b4273afc264843029ed0baf1fc5c8a3e P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:38.102423 18986 leader_election.cc:304] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:38.103415 20204 leader_election.cc:290] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:38.105526 18986 leader_election.cc:304] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:38.107244 20295 raft_consensus.cc:2798] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250111 02:57:38.109148 18616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.126:45115, user_credentials={real_user=slave}} blocked reactor thread for 58599.8us
I20250111 02:57:38.109501 20295 raft_consensus.cc:491] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:38.110092 20295 raft_consensus.cc:3054] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:38.098549 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 70576.8us
I20250111 02:57:38.034173 20182 consensus_queue.cc:1035] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.087s
I20250111 02:57:38.132140 20254 raft_consensus.cc:491] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:38.132660 20254 raft_consensus.cc:513] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.136513 20305 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250111 02:57:38.144449 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:38.069519 19321 ts_tablet_manager.cc:1462] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa: Trace:
0111 02:57:27.273799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:27.273925 (+ 126us) service_pool.cc:225] Handling call
0111 02:57:27.307395 (+ 33470us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0111 02:57:27.307488 (+ 93us) ts_tablet_manager.cc:676] Creating new metadata...
0111 02:57:27.335122 (+ 27634us) tablet_metadata.cc:641] Metadata flushed
0111 02:57:27.353148 (+ 18026us) tablet_replica.cc:206] Creating consensus instance
0111 02:57:27.411270 (+ 58122us) inbound_call.cc:173] Queueing success response
0111 02:57:37.222055 (+9810785us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0111 02:57:37.517694 (+295639us) ts_tablet_manager.cc:1429] Starting tablet replica
0111 02:57:37.518748 (+ 1054us) tablet_replica.cc:260] Starting instrumentation
0111 02:57:37.681691 (+162943us) tablet_replica.cc:279] Starting consensus
Metrics: {"mutex_wait_us":26677,"raft.queue_time_us":29852,"raft.run_cpu_time_us":371,"raft.run_wall_time_us":371,"tablet-open.queue_time_us":9809796}
I20250111 02:57:38.146181 19321 tablet_bootstrap.cc:492] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:38.142839 19330 ts_tablet_manager.cc:1428] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.314s user 0.018s sys 0.001s
I20250111 02:57:38.146822 20254 leader_election.cc:290] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:38.152541 19192 raft_consensus.cc:2463] T b4273afc264843029ed0baf1fc5c8a3e P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:38.153833 18841 leader_election.cc:304] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:38.154425 19330 tablet_bootstrap.cc:492] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:38.154642 20254 raft_consensus.cc:2798] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:38.155117 20254 raft_consensus.cc:695] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 LEADER]: Becoming Leader. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Running, Role: LEADER
I20250111 02:57:38.156071 20254 consensus_queue.cc:237] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:38.160468 19594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 75980.9us
I20250111 02:57:38.171823 19321 tablet_bootstrap.cc:654] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:38.179481 20295 raft_consensus.cc:513] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.185717 20254 consensus_queue.cc:1035] T 0b6687f39d4246b49c0c3eb904b11c1a P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:38.192478 20204 consensus_queue.cc:1035] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:38.197034 19322 tablet_bootstrap.cc:492] T b26e5db463e94d919640a07a2f8574a8 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:38.197533 19322 ts_tablet_manager.cc:1397] T b26e5db463e94d919640a07a2f8574a8 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.352s user 0.016s sys 0.013s
W20250111 02:57:38.183862 18989 leader_election.cc:343] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:38.192183 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 86998.7us
I20250111 02:57:38.215523 18986 leader_election.cc:304] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:38.216341 20294 raft_consensus.cc:2798] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:38.219336 20204 consensus_queue.cc:1035] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:38.216784 20294 raft_consensus.cc:491] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:38.220278 19191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934c900baf1b4000af7d7ec60cbb3591" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:38.220559 20294 raft_consensus.cc:3054] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:38.221002 19191 raft_consensus.cc:3054] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:38.225633 19322 raft_consensus.cc:357] T b26e5db463e94d919640a07a2f8574a8 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.226382 19322 raft_consensus.cc:383] T b26e5db463e94d919640a07a2f8574a8 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:38.226683 19322 raft_consensus.cc:738] T b26e5db463e94d919640a07a2f8574a8 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:38.234297 19191 raft_consensus.cc:2463] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
W20250111 02:57:38.239356 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 55368.2us
I20250111 02:57:38.239411 19322 consensus_queue.cc:260] T b26e5db463e94d919640a07a2f8574a8 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.243275 19322 ts_tablet_manager.cc:1428] T b26e5db463e94d919640a07a2f8574a8 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
I20250111 02:57:38.244285 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:38.245213 19050 raft_consensus.cc:2463] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:38.202972 20182 raft_consensus.cc:2798] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:38.246027 20182 raft_consensus.cc:491] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:38.246481 20182 raft_consensus.cc:3054] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:38.247575 19322 tablet_bootstrap.cc:492] T 934c900baf1b4000af7d7ec60cbb3591 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:38.255530 19321 tablet_bootstrap.cc:492] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:38.204928 20295 leader_election.cc:290] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:38.274716 19322 tablet_bootstrap.cc:654] T 934c900baf1b4000af7d7ec60cbb3591 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:38.287454 19321 ts_tablet_manager.cc:1397] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.141s user 0.012s sys 0.000s
I20250111 02:57:38.291833 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1870cceafe463b8d024e8b065850e0" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:38.292452 19192 raft_consensus.cc:3054] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:38.299459 19192 raft_consensus.cc:2463] T df1870cceafe463b8d024e8b065850e0 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:38.301019 19322 tablet_bootstrap.cc:492] T 934c900baf1b4000af7d7ec60cbb3591 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:38.300695 18900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1870cceafe463b8d024e8b065850e0" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:38.301451 19322 ts_tablet_manager.cc:1397] T 934c900baf1b4000af7d7ec60cbb3591 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.005s
W20250111 02:57:38.311218 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 93074.9us
I20250111 02:57:38.326040 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c669a3ff8aa46fa8b04b86ed3bfde49" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:38.335835 20295 consensus_queue.cc:1035] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:38.338992 20204 raft_consensus.cc:491] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:38.339555 20204 raft_consensus.cc:513] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.341701 20204 leader_election.cc:290] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:38.344249 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b7616fe56f4e5fb49bdde6961fa008" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:38.345721 19133 leader_election.cc:343] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:38.347180 19330 tablet_bootstrap.cc:654] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:38.361845 20182 raft_consensus.cc:513] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.368009 19322 raft_consensus.cc:357] T 934c900baf1b4000af7d7ec60cbb3591 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.368829 19322 raft_consensus.cc:383] T 934c900baf1b4000af7d7ec60cbb3591 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:38.369095 19322 raft_consensus.cc:738] T 934c900baf1b4000af7d7ec60cbb3591 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:38.376026 18842 leader_election.cc:304] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters: d0cedab82b154d62b78809edc6f28bfa
W20250111 02:57:38.370625 18989 consensus_peers.cc:487] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:38.377027 18989 leader_election.cc:343] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:38.367594 20294 raft_consensus.cc:513] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.364553 19321 raft_consensus.cc:357] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:38.387441 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 60195.4us
I20250111 02:57:38.388464 20254 raft_consensus.cc:2798] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 02:57:38.388906 20254 raft_consensus.cc:491] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0cedab82b154d62b78809edc6f28bfa)
I20250111 02:57:38.389361 20254 raft_consensus.cc:3054] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:57:38.385280 19321 raft_consensus.cc:383] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:38.399482 20254 raft_consensus.cc:513] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.393663 19321 raft_consensus.cc:738] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:38.402956 18986 leader_election.cc:304] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:38.404582 20229 raft_consensus.cc:2798] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:38.405284 20229 raft_consensus.cc:695] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:38.406312 20294 leader_election.cc:290] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:38.406639 20229 consensus_queue.cc:237] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.409044 20182 leader_election.cc:290] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:38.418458 20297 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:38.419040 20297 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:38.421571 20297 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:38.421553 19322 consensus_queue.cc:260] T 934c900baf1b4000af7d7ec60cbb3591 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.423367 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:38.425189 18986 leader_election.cc:304] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:38.427158 20148 raft_consensus.cc:2798] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:38.427722 20148 raft_consensus.cc:695] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:38.432018 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7145efe5c0094b70be2cd6481b770248" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:38.439334 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42230a0bc94249fda91cf79fba48f024" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:38.441555 20254 leader_election.cc:290] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:38.442267 19191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:38.428761 20148 consensus_queue.cc:237] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.484108 19322 ts_tablet_manager.cc:1428] T 934c900baf1b4000af7d7ec60cbb3591 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.182s user 0.009s sys 0.000s
I20250111 02:57:38.494840 20182 raft_consensus.cc:491] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:38.412081 19321 consensus_queue.cc:260] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.495460 20182 raft_consensus.cc:513] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.520547 20182 leader_election.cc:290] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:38.524533 20254 raft_consensus.cc:491] T 934c900baf1b4000af7d7ec60cbb3591 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:38.525112 20254 raft_consensus.cc:513] T 934c900baf1b4000af7d7ec60cbb3591 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.493811 20294 consensus_queue.cc:1035] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
W20250111 02:57:38.528191 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 58965.3us
I20250111 02:57:38.528914 19191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934c900baf1b4000af7d7ec60cbb3591" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:38.529940 19191 raft_consensus.cc:2388] T 934c900baf1b4000af7d7ec60cbb3591 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
W20250111 02:57:38.531098 19131 leader_election.cc:343] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:38.531553 19131 leader_election.cc:304] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:38.534257 20310 raft_consensus.cc:2743] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:38.543969 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7145efe5c0094b70be2cd6481b770248" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:38.544674 19194 raft_consensus.cc:3054] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:38.546543 19330 tablet_bootstrap.cc:492] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:38.546957 19330 ts_tablet_manager.cc:1397] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.393s user 0.005s sys 0.008s
I20250111 02:57:38.548743 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2009d6f54e2842e8be3af2cc9147dc2c" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:38.549628 19193 raft_consensus.cc:2463] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:38.552726 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42230a0bc94249fda91cf79fba48f024" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:38.553361 19197 raft_consensus.cc:3054] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:38.557191 20151 consensus_queue.cc:1035] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:38.558944 20254 leader_election.cc:290] T 934c900baf1b4000af7d7ec60cbb3591 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:38.562723 19322 tablet_bootstrap.cc:492] T d837bd001aa649e9b3ca94f7cdd9f0fe P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:38.564635 19330 raft_consensus.cc:357] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.565326 19330 raft_consensus.cc:383] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:38.565583 19330 raft_consensus.cc:738] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
W20250111 02:57:38.570505 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 74297.6us
W20250111 02:57:38.582403 18989 leader_election.cc:343] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:38.585309 19330 consensus_queue.cc:260] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:38.585733 18989 leader_election.cc:343] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:38.589994 18986 leader_election.cc:304] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:38.592149 19197 raft_consensus.cc:2463] T 42230a0bc94249fda91cf79fba48f024 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:38.597208 20294 raft_consensus.cc:2798] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:38.597658 20294 raft_consensus.cc:491] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:38.598057 20294 raft_consensus.cc:3054] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:38.598879 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b7616fe56f4e5fb49bdde6961fa008" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:38.599889 19330 ts_tablet_manager.cc:1428] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.053s user 0.002s sys 0.004s
I20250111 02:57:38.608810 19322 tablet_bootstrap.cc:654] T d837bd001aa649e9b3ca94f7cdd9f0fe P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:38.592803 19194 raft_consensus.cc:2463] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
W20250111 02:57:38.603860 19131 leader_election.cc:343] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:38.612248 19131 leader_election.cc:304] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:38.613497 20151 raft_consensus.cc:2743] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:38.625444 20254 raft_consensus.cc:491] T adf61f6cb63d4e8ea46b440e7ecce00d P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:38.626077 20254 raft_consensus.cc:513] T adf61f6cb63d4e8ea46b440e7ecce00d P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.628764 20254 leader_election.cc:290] T adf61f6cb63d4e8ea46b440e7ecce00d P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:38.629555 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf61f6cb63d4e8ea46b440e7ecce00d" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:38.631477 18910 raft_consensus.cc:3054] T adf61f6cb63d4e8ea46b440e7ecce00d P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:38.631515 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2009d6f54e2842e8be3af2cc9147dc2c" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:38.635870 19330 tablet_bootstrap.cc:492] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:38.636198 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934c900baf1b4000af7d7ec60cbb3591" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:38.638562 18989 leader_election.cc:343] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:38.642602 18842 leader_election.cc:304] T 934c900baf1b4000af7d7ec60cbb3591 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:38.640460 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:38.644645 20254 raft_consensus.cc:3054] T 934c900baf1b4000af7d7ec60cbb3591 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:38.653935 18842 leader_election.cc:304] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:38.620167 19321 ts_tablet_manager.cc:1428] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.332s user 0.016s sys 0.015s
I20250111 02:57:38.654582 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf61f6cb63d4e8ea46b440e7ecce00d" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:38.654841 20292 raft_consensus.cc:2743] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250111 02:57:38.624226 18986 leader_election.cc:304] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
W20250111 02:57:38.668149 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 44024.2us
I20250111 02:57:38.671885 20295 raft_consensus.cc:2798] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:38.673470 20295 raft_consensus.cc:695] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:38.650821 20294 raft_consensus.cc:513] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.675870 18842 leader_election.cc:304] T adf61f6cb63d4e8ea46b440e7ecce00d P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:38.674664 20295 consensus_queue.cc:237] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.677570 19330 tablet_bootstrap.cc:654] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:38.678323 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2009d6f54e2842e8be3af2cc9147dc2c" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:38.673440 18986 leader_election.cc:304] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:38.680867 19321 tablet_bootstrap.cc:492] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:38.683032 20322 raft_consensus.cc:2798] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
W20250111 02:57:38.683357 18989 leader_election.cc:343] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:38.684118 20322 raft_consensus.cc:695] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:38.715956 20254 raft_consensus.cc:2743] T 934c900baf1b4000af7d7ec60cbb3591 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:38.695030 20294 leader_election.cc:290] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:38.685192 20322 consensus_queue.cc:237] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.753682 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2009d6f54e2842e8be3af2cc9147dc2c" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:38.754571 19192 raft_consensus.cc:3054] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:38.757715 18907 raft_consensus.cc:3054] T b26e5db463e94d919640a07a2f8574a8 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:38.768648 19322 tablet_bootstrap.cc:492] T d837bd001aa649e9b3ca94f7cdd9f0fe P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:38.769174 19322 ts_tablet_manager.cc:1397] T d837bd001aa649e9b3ca94f7cdd9f0fe P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.245s user 0.017s sys 0.010s
I20250111 02:57:38.773702 19322 raft_consensus.cc:357] T d837bd001aa649e9b3ca94f7cdd9f0fe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.774528 19322 raft_consensus.cc:383] T d837bd001aa649e9b3ca94f7cdd9f0fe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:38.774828 19322 raft_consensus.cc:738] T d837bd001aa649e9b3ca94f7cdd9f0fe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:38.775578 19322 consensus_queue.cc:260] T d837bd001aa649e9b3ca94f7cdd9f0fe P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.778517 20292 raft_consensus.cc:2743] T adf61f6cb63d4e8ea46b440e7ecce00d P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:38.787549 19192 raft_consensus.cc:2463] T 2009d6f54e2842e8be3af2cc9147dc2c P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:38.815357 18986 leader_election.cc:304] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:38.817673 20294 raft_consensus.cc:2798] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:38.818284 20294 raft_consensus.cc:695] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
W20250111 02:57:38.839246 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 54720.2us
I20250111 02:57:38.840871 19322 ts_tablet_manager.cc:1428] T d837bd001aa649e9b3ca94f7cdd9f0fe P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.071s user 0.006s sys 0.001s
I20250111 02:57:38.822252 20294 consensus_queue.cc:237] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.840780 20182 raft_consensus.cc:491] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:38.854547 20182 raft_consensus.cc:513] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.862357 19321 tablet_bootstrap.cc:654] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:38.869560 19330 tablet_bootstrap.cc:492] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:38.870057 19330 ts_tablet_manager.cc:1397] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.234s user 0.006s sys 0.013s
I20250111 02:57:38.871027 19322 tablet_bootstrap.cc:492] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:38.873504 19330 raft_consensus.cc:357] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.882200 20296 raft_consensus.cc:491] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:38.882860 20296 raft_consensus.cc:513] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.885710 20296 leader_election.cc:290] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:38.887598 20182 leader_election.cc:290] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:38.889667 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb87813c28742edad99470c84269ef4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:38.889060 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873325cece64db6a673d4fe08eb7409" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:38.896426 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873325cece64db6a673d4fe08eb7409" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:38.897186 19192 raft_consensus.cc:2463] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:38.900535 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb87813c28742edad99470c84269ef4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:38.901546 19192 raft_consensus.cc:2463] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
W20250111 02:57:38.898451 18989 leader_election.cc:343] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:38.903044 18989 leader_election.cc:343] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:38.874284 19330 raft_consensus.cc:383] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:38.908370 19330 raft_consensus.cc:738] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:38.913823 18986 leader_election.cc:304] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:38.910851 19330 consensus_queue.cc:260] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.915092 20322 raft_consensus.cc:2798] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:38.915561 20322 raft_consensus.cc:491] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:38.915997 20322 raft_consensus.cc:3054] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:38.920409 19322 tablet_bootstrap.cc:654] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:38.921289 18986 leader_election.cc:304] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:38.923215 20302 raft_consensus.cc:491] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:38.923817 20302 raft_consensus.cc:513] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.927232 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaab428f798a44f69c80e5792d56b920" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:38.928880 19133 leader_election.cc:343] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:38.930140 20302 leader_election.cc:290] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:38.931499 19330 ts_tablet_manager.cc:1428] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.061s user 0.002s sys 0.005s
I20250111 02:57:38.939285 20182 raft_consensus.cc:2798] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:38.939822 20182 raft_consensus.cc:491] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:38.940619 20182 raft_consensus.cc:3054] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:38.981128 19322 tablet_bootstrap.cc:492] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:38.981647 19322 ts_tablet_manager.cc:1397] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.111s user 0.005s sys 0.007s
I20250111 02:57:38.984666 19322 raft_consensus.cc:357] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:38.985428 19322 raft_consensus.cc:383] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:38.985687 19322 raft_consensus.cc:738] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:38.986485 19322 consensus_queue.cc:260] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.001806 19330 tablet_bootstrap.cc:492] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:39.008695 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaab428f798a44f69c80e5792d56b920" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:39.017225 20292 consensus_queue.cc:1035] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250111 02:57:39.024677 18614 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b4273afc264843029ed0baf1fc5c8a3e (table default.Table_4e3a79a5d2184687a8c51fd2702c5fd0 [id=73c59c0d9864497fbc9f19e265d8283f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b4273afc264843029ed0baf1fc5c8a3e consensus state has no leader
I20250111 02:57:39.037504 19321 tablet_bootstrap.cc:492] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:39.037946 19321 ts_tablet_manager.cc:1397] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.357s user 0.018s sys 0.003s
I20250111 02:57:39.040932 19321 raft_consensus.cc:357] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.041733 19321 raft_consensus.cc:383] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:39.042001 19321 raft_consensus.cc:738] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:39.043025 20292 consensus_queue.cc:1035] T b4273afc264843029ed0baf1fc5c8a3e P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250111 02:57:39.046723 19131 leader_election.cc:343] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:39.047202 19131 leader_election.cc:304] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:39.047968 20151 raft_consensus.cc:2743] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:39.052328 19322 ts_tablet_manager.cc:1428] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.070s user 0.008s sys 0.000s
I20250111 02:57:39.053445 19322 tablet_bootstrap.cc:492] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:39.064659 20322 raft_consensus.cc:513] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.072156 20344 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
W20250111 02:57:39.075314 18693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 112806us
I20250111 02:57:39.076927 20344 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:39.081686 19330 tablet_bootstrap.cc:654] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:39.101197 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873325cece64db6a673d4fe08eb7409" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:39.102802 18989 leader_election.cc:343] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:39.107208 20322 leader_election.cc:290] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:39.111673 20344 leader_election.cc:290] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8784b9603f8544b79373158aace063af (127.18.20.125:40359), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:39.113405 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:39.042630 19321 consensus_queue.cc:260] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.121193 19321 ts_tablet_manager.cc:1428] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.083s user 0.002s sys 0.003s
I20250111 02:57:39.122241 19321 tablet_bootstrap.cc:492] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:39.123304 20182 raft_consensus.cc:513] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.125501 20182 leader_election.cc:290] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:39.130713 19322 tablet_bootstrap.cc:654] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:39.156374 19321 tablet_bootstrap.cc:654] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:39.161751 19322 tablet_bootstrap.cc:492] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:39.162214 19322 ts_tablet_manager.cc:1397] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.109s user 0.006s sys 0.003s
I20250111 02:57:39.169052 19321 tablet_bootstrap.cc:492] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:39.166769 19322 raft_consensus.cc:357] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.179697 19322 raft_consensus.cc:383] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:39.188653 19322 raft_consensus.cc:738] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:39.189318 19322 consensus_queue.cc:260] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.191429 19321 ts_tablet_manager.cc:1397] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.069s user 0.009s sys 0.003s
I20250111 02:57:39.194610 20292 raft_consensus.cc:491] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:39.195140 20292 raft_consensus.cc:513] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.197247 20292 leader_election.cc:290] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:39.198995 19191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873325cece64db6a673d4fe08eb7409" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:39.199714 19191 raft_consensus.cc:2463] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:39.201572 18841 leader_election.cc:304] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:39.202589 20292 raft_consensus.cc:2798] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:39.202632 19321 raft_consensus.cc:357] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:39.203002 20292 raft_consensus.cc:491] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:39.203372 19321 raft_consensus.cc:383] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:39.203536 20292 raft_consensus.cc:3054] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:39.203644 19321 raft_consensus.cc:738] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:39.204313 19321 consensus_queue.cc:260] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:39.206708 19321 ts_tablet_manager.cc:1428] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20250111 02:57:39.207777 19322 ts_tablet_manager.cc:1428] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.045s user 0.008s sys 0.000s
I20250111 02:57:39.208873 19322 tablet_bootstrap.cc:492] T 7145efe5c0094b70be2cd6481b770248 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:39.219118 19322 tablet_bootstrap.cc:654] T 7145efe5c0094b70be2cd6481b770248 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:39.229346 19321 tablet_bootstrap.cc:492] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:39.211052 20292 raft_consensus.cc:513] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.238332 19321 tablet_bootstrap.cc:654] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:39.262125 19191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873325cece64db6a673d4fe08eb7409" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:39.262984 19191 raft_consensus.cc:3054] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:39.264657 19330 tablet_bootstrap.cc:492] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:39.265480 19330 ts_tablet_manager.cc:1397] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.264s user 0.006s sys 0.019s
I20250111 02:57:39.269483 18907 raft_consensus.cc:1235] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Rejecting Update request from peer d0cedab82b154d62b78809edc6f28bfa for earlier term 1. Current term is 2. Ops: []
I20250111 02:57:39.271298 20302 consensus_queue.cc:1046] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:57:39.272261 20302 raft_consensus.cc:3049] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:57:39.272619 20302 raft_consensus.cc:738] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:39.272754 19191 raft_consensus.cc:2463] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:39.273485 20302 consensus_queue.cc:260] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.274899 20302 raft_consensus.cc:3054] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:57:39.283088 20254 raft_consensus.cc:491] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:39.283666 20254 raft_consensus.cc:513] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.286000 20254 leader_election.cc:290] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:39.295711 20322 raft_consensus.cc:491] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:39.295832 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933828fd6fb041e9a4889f4ea8a3c5ff" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:39.296919 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb87813c28742edad99470c84269ef4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:39.296694 20322 raft_consensus.cc:513] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.297575 19194 raft_consensus.cc:3054] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:39.294687 19330 raft_consensus.cc:357] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.298323 19330 raft_consensus.cc:383] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:39.298653 19330 raft_consensus.cc:738] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:39.299540 20322 leader_election.cc:290] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:39.299386 19330 consensus_queue.cc:260] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.301242 19191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873325cece64db6a673d4fe08eb7409" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:39.302142 19191 raft_consensus.cc:2388] T 2873325cece64db6a673d4fe08eb7409 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:39.303826 19330 ts_tablet_manager.cc:1428] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.038s user 0.010s sys 0.003s
I20250111 02:57:39.304374 20292 leader_election.cc:290] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:39.305672 19330 tablet_bootstrap.cc:492] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:39.319849 19194 raft_consensus.cc:2463] T 8eb87813c28742edad99470c84269ef4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:39.321940 20292 raft_consensus.cc:491] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 54628fbcec2f4359aa9e26a734d4d29f)
I20250111 02:57:39.322458 20292 raft_consensus.cc:513] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.324541 20292 leader_election.cc:290] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:39.345008 18841 leader_election.cc:343] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Not found: Tablet not found: 4b302fcba27244c193586bc25fc9305f
I20250111 02:57:39.344511 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873325cece64db6a673d4fe08eb7409" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:39.346253 19050 raft_consensus.cc:2388] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:39.347831 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933828fd6fb041e9a4889f4ea8a3c5ff" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:39.348413 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb87813c28742edad99470c84269ef4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:39.349517 18910 raft_consensus.cc:3054] T d837bd001aa649e9b3ca94f7cdd9f0fe P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:39.359616 18989 leader_election.cc:343] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:39.362149 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6da575f052d4b20a8e41f0bd9dab7e4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:39.362728 18842 leader_election.cc:304] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:39.363509 20254 raft_consensus.cc:2743] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:39.368892 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873325cece64db6a673d4fe08eb7409" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:39.369752 19052 raft_consensus.cc:2388] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:39.376852 18986 leader_election.cc:304] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:39.377853 20294 raft_consensus.cc:2798] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:39.378324 18986 leader_election.cc:304] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:39.378341 20294 raft_consensus.cc:695] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:39.379436 19321 tablet_bootstrap.cc:492] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:39.379873 19321 ts_tablet_manager.cc:1397] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.151s user 0.015s sys 0.011s
I20250111 02:57:39.380475 20314 raft_consensus.cc:2798] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:39.381002 20314 raft_consensus.cc:695] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:39.379459 20294 consensus_queue.cc:237] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.381973 20314 consensus_queue.cc:237] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.382766 19321 raft_consensus.cc:357] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:39.389313 19321 raft_consensus.cc:383] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:39.389622 19321 raft_consensus.cc:738] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:39.390241 19321 consensus_queue.cc:260] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:39.395008 18842 leader_election.cc:304] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:39.395761 20254 raft_consensus.cc:2743] T 2873325cece64db6a673d4fe08eb7409 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250111 02:57:39.396956 18842 leader_election.cc:343] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: 4b302fcba27244c193586bc25fc9305f
I20250111 02:57:39.397352 18842 leader_election.cc:304] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:39.398051 20254 raft_consensus.cc:2743] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:39.402675 19321 ts_tablet_manager.cc:1428] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.023s user 0.004s sys 0.003s
I20250111 02:57:39.404980 19330 tablet_bootstrap.cc:654] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:39.414026 19322 tablet_bootstrap.cc:492] T 7145efe5c0094b70be2cd6481b770248 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:39.414508 19322 ts_tablet_manager.cc:1397] T 7145efe5c0094b70be2cd6481b770248 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.206s user 0.016s sys 0.018s
I20250111 02:57:39.417554 19322 raft_consensus.cc:357] T 7145efe5c0094b70be2cd6481b770248 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.418273 19322 raft_consensus.cc:383] T 7145efe5c0094b70be2cd6481b770248 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:39.418525 19322 raft_consensus.cc:738] T 7145efe5c0094b70be2cd6481b770248 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:39.424263 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6da575f052d4b20a8e41f0bd9dab7e4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:39.424811 19321 tablet_bootstrap.cc:492] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:39.424988 19194 raft_consensus.cc:2463] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:39.430506 19321 tablet_bootstrap.cc:654] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:39.427841 18989 leader_election.cc:343] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:39.443681 19322 consensus_queue.cc:260] T 7145efe5c0094b70be2cd6481b770248 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.448767 19322 ts_tablet_manager.cc:1428] T 7145efe5c0094b70be2cd6481b770248 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.034s user 0.009s sys 0.000s
I20250111 02:57:39.451536 19321 tablet_bootstrap.cc:492] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:39.451956 19321 ts_tablet_manager.cc:1397] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.012s
I20250111 02:57:39.454753 19321 raft_consensus.cc:357] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:39.455730 19321 raft_consensus.cc:383] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:39.456018 19321 raft_consensus.cc:738] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:39.458106 19321 consensus_queue.cc:260] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:39.465777 18910 raft_consensus.cc:3054] T 933828fd6fb041e9a4889f4ea8a3c5ff P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:39.466892 19321 ts_tablet_manager.cc:1428] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250111 02:57:39.477461 18986 leader_election.cc:304] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:39.481251 20046 raft_consensus.cc:2798] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:39.481676 20046 raft_consensus.cc:491] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:39.481999 20046 raft_consensus.cc:3054] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:39.487882 20351 log.cc:926] Time spent T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f: Append to log took a long time: real 0.092s user 0.001s sys 0.000s
I20250111 02:57:39.488692 19330 tablet_bootstrap.cc:492] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:39.489099 19330 ts_tablet_manager.cc:1397] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.184s user 0.008s sys 0.004s
I20250111 02:57:39.490096 20302 raft_consensus.cc:491] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:39.490581 20302 raft_consensus.cc:513] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:39.492810 19322 tablet_bootstrap.cc:492] T 2009d6f54e2842e8be3af2cc9147dc2c P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:39.492683 20302 leader_election.cc:290] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:39.494732 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f15096ea484b499fbdecec74eb6f66" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:39.496376 19133 leader_election.cc:343] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:39.476806 18989 consensus_peers.cc:487] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:39.498790 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 45631.1us
I20250111 02:57:39.500864 20302 raft_consensus.cc:491] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:39.501389 20302 raft_consensus.cc:513] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:39.503630 20352 log.cc:926] Time spent T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
I20250111 02:57:39.505393 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3514d795a0f24adcaa2accc0cf05b142" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:39.505527 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f15096ea484b499fbdecec74eb6f66" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:39.506855 19133 leader_election.cc:343] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:39.508368 19131 leader_election.cc:343] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:39.508780 19131 leader_election.cc:304] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:39.510185 20151 raft_consensus.cc:2743] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:39.510775 20229 consensus_queue.cc:1035] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:39.515079 20302 leader_election.cc:290] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:39.516047 20321 raft_consensus.cc:491] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:39.517932 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3514d795a0f24adcaa2accc0cf05b142" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:39.524221 19330 raft_consensus.cc:357] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.524957 19330 raft_consensus.cc:383] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:39.525249 19330 raft_consensus.cc:738] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:39.525852 19330 consensus_queue.cc:260] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.516544 20321 raft_consensus.cc:513] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.527823 19321 tablet_bootstrap.cc:492] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:39.528733 20321 leader_election.cc:290] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:39.533285 19131 leader_election.cc:343] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:39.533793 19131 leader_election.cc:304] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:39.535208 20302 raft_consensus.cc:2743] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:39.536160 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca35147034446b3882b5244923bddc7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:39.536809 19194 raft_consensus.cc:2463] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:39.550266 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca35147034446b3882b5244923bddc7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:39.554647 19321 tablet_bootstrap.cc:654] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:39.565160 19322 tablet_bootstrap.cc:654] T 2009d6f54e2842e8be3af2cc9147dc2c P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:39.567716 19330 ts_tablet_manager.cc:1428] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.078s user 0.005s sys 0.003s
I20250111 02:57:39.569108 19330 tablet_bootstrap.cc:492] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
W20250111 02:57:39.574767 18989 consensus_peers.cc:487] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:39.575850 18989 leader_election.cc:343] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:39.578922 18986 leader_election.cc:304] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:39.579849 20295 consensus_queue.cc:1035] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:39.581087 20295 raft_consensus.cc:2798] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:39.581496 20295 raft_consensus.cc:491] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:39.581892 20295 raft_consensus.cc:3054] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:39.616266 18693 leader_election.cc:336] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.539s (SENT)
I20250111 02:57:39.616724 18693 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:39.554721 20046 raft_consensus.cc:513] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.629686 20046 leader_election.cc:290] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:39.634743 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6da575f052d4b20a8e41f0bd9dab7e4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:39.635382 19197 raft_consensus.cc:3054] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:39.636597 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6da575f052d4b20a8e41f0bd9dab7e4" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:39.639779 18989 leader_election.cc:343] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:39.643988 20357 consensus_queue.cc:1035] T 42230a0bc94249fda91cf79fba48f024 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:39.643333 20364 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250111 02:57:39.646526 19197 raft_consensus.cc:2463] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:39.657555 19321 tablet_bootstrap.cc:492] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:39.671440 18986 leader_election.cc:304] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:39.672510 20357 raft_consensus.cc:2798] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:39.673059 20357 raft_consensus.cc:695] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:39.674052 20357 consensus_queue.cc:237] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.682605 19321 ts_tablet_manager.cc:1397] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.155s user 0.008s sys 0.007s
I20250111 02:57:39.685479 19330 tablet_bootstrap.cc:654] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:39.731722 19594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 68691.9us
I20250111 02:57:39.760215 20295 raft_consensus.cc:513] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.692277 19321 raft_consensus.cc:357] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:39.770139 19321 raft_consensus.cc:383] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:39.770465 19321 raft_consensus.cc:738] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:39.771194 19321 consensus_queue.cc:260] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:39.761276 20321 consensus_queue.cc:1035] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:39.788771 20295 leader_election.cc:290] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:39.794301 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca35147034446b3882b5244923bddc7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:39.795873 18989 leader_election.cc:343] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:39.796145 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca35147034446b3882b5244923bddc7" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:39.823523 19194 raft_consensus.cc:3054] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:39.835875 19322 tablet_bootstrap.cc:492] T 2009d6f54e2842e8be3af2cc9147dc2c P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:39.837183 19322 ts_tablet_manager.cc:1397] T 2009d6f54e2842e8be3af2cc9147dc2c P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.345s user 0.015s sys 0.002s
I20250111 02:57:39.838781 20254 raft_consensus.cc:491] T 7145efe5c0094b70be2cd6481b770248 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:39.839329 20254 raft_consensus.cc:513] T 7145efe5c0094b70be2cd6481b770248 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.841560 20254 leader_election.cc:290] T 7145efe5c0094b70be2cd6481b770248 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:39.842618 18989 consensus_peers.cc:487] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:39.843379 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7145efe5c0094b70be2cd6481b770248" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:39.846310 19194 raft_consensus.cc:2463] T aca35147034446b3882b5244923bddc7 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:39.847808 18986 leader_election.cc:304] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:39.849633 20296 raft_consensus.cc:2798] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:39.850240 20296 raft_consensus.cc:695] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:39.851261 20296 consensus_queue.cc:237] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.864928 19321 ts_tablet_manager.cc:1428] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.182s user 0.009s sys 0.004s
I20250111 02:57:39.868204 19330 tablet_bootstrap.cc:492] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:39.868767 19330 ts_tablet_manager.cc:1397] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.300s user 0.016s sys 0.008s
I20250111 02:57:39.872530 19322 raft_consensus.cc:357] T 2009d6f54e2842e8be3af2cc9147dc2c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.873314 19322 raft_consensus.cc:383] T 2009d6f54e2842e8be3af2cc9147dc2c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:39.873607 19322 raft_consensus.cc:738] T 2009d6f54e2842e8be3af2cc9147dc2c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:39.920672 19330 raft_consensus.cc:357] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.921469 19330 raft_consensus.cc:383] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:39.921751 19330 raft_consensus.cc:738] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:39.878604 19322 consensus_queue.cc:260] T 2009d6f54e2842e8be3af2cc9147dc2c P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.930573 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7145efe5c0094b70be2cd6481b770248" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:39.940646 19197 raft_consensus.cc:2388] T 7145efe5c0094b70be2cd6481b770248 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:39.943940 20310 raft_consensus.cc:491] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:39.944456 20310 raft_consensus.cc:513] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:39.946461 20310 leader_election.cc:290] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:39.955271 20357 consensus_queue.cc:1035] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:39.957368 20310 raft_consensus.cc:491] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:39.957828 20310 raft_consensus.cc:513] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.958330 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c57d85390324bbc9ea30a709faec01c" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:39.962836 19322 ts_tablet_manager.cc:1428] T 2009d6f54e2842e8be3af2cc9147dc2c P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.125s user 0.008s sys 0.002s
I20250111 02:57:39.963718 20310 leader_election.cc:290] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:39.966281 18910 raft_consensus.cc:3054] T 7145efe5c0094b70be2cd6481b770248 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:39.966898 19131 leader_election.cc:343] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:39.971553 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f561cfd4f34426b7e42027e53b3ff1" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:39.974416 18841 leader_election.cc:304] T 7145efe5c0094b70be2cd6481b770248 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:39.922796 19330 consensus_queue.cc:260] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:39.977785 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c57d85390324bbc9ea30a709faec01c" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:39.977824 18897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f561cfd4f34426b7e42027e53b3ff1" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:39.979586 19131 leader_election.cc:343] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:39.980737 19133 leader_election.cc:343] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:39.980984 19330 ts_tablet_manager.cc:1428] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.111s user 0.006s sys 0.002s
I20250111 02:57:39.981242 19133 leader_election.cc:304] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
W20250111 02:57:39.982420 19133 leader_election.cc:343] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:39.983158 19133 leader_election.cc:304] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:40.006716 19322 tablet_bootstrap.cc:492] T df1870cceafe463b8d024e8b065850e0 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:40.009248 19330 tablet_bootstrap.cc:492] T 36f71f8dd2894d40b3bfdd20c1453787 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:40.023526 20357 consensus_queue.cc:1035] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250111 02:57:40.035023 19133 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 52635.3us
I20250111 02:57:39.984583 20302 raft_consensus.cc:2743] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:40.035957 20302 raft_consensus.cc:2743] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:40.039500 20254 raft_consensus.cc:2743] T 7145efe5c0094b70be2cd6481b770248 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:40.043359 20229 consensus_queue.cc:1035] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:40.045857 19330 tablet_bootstrap.cc:654] T 36f71f8dd2894d40b3bfdd20c1453787 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:40.074118 19322 tablet_bootstrap.cc:654] T df1870cceafe463b8d024e8b065850e0 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:40.099946 20384 tablet.cc:1721] T cc7158a7fc624fa4bc816ac3721c610d P d0cedab82b154d62b78809edc6f28bfa: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250111 02:57:40.102731 20382 tablet.cc:1721] T cc7158a7fc624fa4bc816ac3721c610d P 54628fbcec2f4359aa9e26a734d4d29f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250111 02:57:40.136886 19322 tablet_bootstrap.cc:492] T df1870cceafe463b8d024e8b065850e0 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:40.137307 19322 ts_tablet_manager.cc:1397] T df1870cceafe463b8d024e8b065850e0 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.131s user 0.013s sys 0.001s
I20250111 02:57:40.141970 19330 tablet_bootstrap.cc:492] T 36f71f8dd2894d40b3bfdd20c1453787 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:40.142431 19330 ts_tablet_manager.cc:1397] T 36f71f8dd2894d40b3bfdd20c1453787 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.133s user 0.006s sys 0.005s
I20250111 02:57:40.144946 19330 raft_consensus.cc:357] T 36f71f8dd2894d40b3bfdd20c1453787 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.145682 19330 raft_consensus.cc:383] T 36f71f8dd2894d40b3bfdd20c1453787 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:40.145974 19330 raft_consensus.cc:738] T 36f71f8dd2894d40b3bfdd20c1453787 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:40.149287 19322 raft_consensus.cc:357] T df1870cceafe463b8d024e8b065850e0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.156363 19321 tablet_bootstrap.cc:492] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:40.156657 20357 raft_consensus.cc:491] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:40.157199 20357 raft_consensus.cc:513] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.160360 18897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01771c56ea5498a876d3bb865c752a2" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:40.161979 18989 leader_election.cc:343] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:40.163640 20357 leader_election.cc:290] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:40.166627 19321 tablet_bootstrap.cc:654] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:40.168712 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01771c56ea5498a876d3bb865c752a2" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:40.169385 19197 raft_consensus.cc:2463] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:40.171005 18986 leader_election.cc:304] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:40.175936 20392 tablet.cc:1721] T cc7158a7fc624fa4bc816ac3721c610d P 8221604733fe4fc0a1fc32de8d6af71c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250111 02:57:40.183699 20296 raft_consensus.cc:2798] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:40.184211 20296 raft_consensus.cc:491] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:40.184576 20296 raft_consensus.cc:3054] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:40.192608 19330 consensus_queue.cc:260] T 36f71f8dd2894d40b3bfdd20c1453787 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.195250 19322 raft_consensus.cc:383] T df1870cceafe463b8d024e8b065850e0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:40.195739 19322 raft_consensus.cc:738] T df1870cceafe463b8d024e8b065850e0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:40.196450 19322 consensus_queue.cc:260] T df1870cceafe463b8d024e8b065850e0 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.203752 20310 raft_consensus.cc:491] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:40.204263 20310 raft_consensus.cc:513] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:40.206236 20310 leader_election.cc:290] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:40.213970 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dd0bccf6854d37a7ef95b5ab06308e" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:40.217346 19133 leader_election.cc:343] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:40.194989 19330 ts_tablet_manager.cc:1428] T 36f71f8dd2894d40b3bfdd20c1453787 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.052s user 0.002s sys 0.005s
I20250111 02:57:40.207935 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dd0bccf6854d37a7ef95b5ab06308e" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:40.224609 19131 leader_election.cc:343] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:40.225041 19131 leader_election.cc:304] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:40.226316 20310 raft_consensus.cc:2743] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:40.228374 19321 tablet_bootstrap.cc:492] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:40.228850 19321 ts_tablet_manager.cc:1397] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.005s
I20250111 02:57:40.198921 19322 ts_tablet_manager.cc:1428] T df1870cceafe463b8d024e8b065850e0 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.061s user 0.003s sys 0.001s
I20250111 02:57:40.240248 19330 tablet_bootstrap.cc:492] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:40.245004 19322 tablet_bootstrap.cc:492] T 42230a0bc94249fda91cf79fba48f024 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:40.247259 19321 raft_consensus.cc:357] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:40.248063 19321 raft_consensus.cc:383] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:40.248356 19321 raft_consensus.cc:738] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:40.249073 19321 consensus_queue.cc:260] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:40.251325 18615 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet fa86b1e6eca7403c9a9c52592b8fb636 (table default.Table_4b9e85e01e1843ce831205177cbb2e0f [id=0a118da8c5d0408980cbd0b33e0c2ed0], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet fa86b1e6eca7403c9a9c52592b8fb636 consensus state has no leader
I20250111 02:57:40.268618 20296 raft_consensus.cc:513] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:40.282385 19594 leader_election.cc:336] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.548s (SENT)
I20250111 02:57:40.282380 18897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01771c56ea5498a876d3bb865c752a2" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:40.282851 19594 leader_election.cc:304] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727; no voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:40.293188 19322 tablet_bootstrap.cc:654] T 42230a0bc94249fda91cf79fba48f024 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:40.312067 19321 ts_tablet_manager.cc:1428] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.083s user 0.007s sys 0.000s
I20250111 02:57:40.323206 19321 tablet_bootstrap.cc:492] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
W20250111 02:57:40.283697 18989 leader_election.cc:343] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:40.329358 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 45699.6us
I20250111 02:57:40.330178 20394 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250111 02:57:40.330819 20357 consensus_queue.cc:1035] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:40.331898 20296 leader_election.cc:290] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:40.331032 19594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 48697.8us
I20250111 02:57:40.332144 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01771c56ea5498a876d3bb865c752a2" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:40.332718 19197 raft_consensus.cc:3054] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:40.339363 19197 raft_consensus.cc:2463] T e01771c56ea5498a876d3bb865c752a2 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:40.341212 18986 leader_election.cc:304] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:40.343468 20358 raft_consensus.cc:2798] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:40.343941 20358 raft_consensus.cc:695] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:40.344890 20358 consensus_queue.cc:237] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.362229 20302 raft_consensus.cc:491] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:40.362723 20302 raft_consensus.cc:513] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:40.364800 20302 leader_election.cc:290] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:40.369434 18897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56fb0bf949904208af003f08caae7594" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:40.371685 19133 leader_election.cc:343] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:40.375466 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56fb0bf949904208af003f08caae7594" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:40.275084 19330 tablet_bootstrap.cc:654] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:40.377439 19131 leader_election.cc:343] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:40.377836 19131 leader_election.cc:304] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:40.379573 20151 raft_consensus.cc:2743] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:40.394484 20295 raft_consensus.cc:491] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:40.395016 20295 raft_consensus.cc:513] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.397384 20295 leader_election.cc:290] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:40.398162 18897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa86b1e6eca7403c9a9c52592b8fb636" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:40.398379 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa86b1e6eca7403c9a9c52592b8fb636" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:40.399191 19197 raft_consensus.cc:2463] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
W20250111 02:57:40.399802 18989 leader_election.cc:343] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:40.400493 18986 leader_election.cc:304] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:40.401346 20295 raft_consensus.cc:2798] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:40.401753 20295 raft_consensus.cc:491] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:40.402127 20295 raft_consensus.cc:3054] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:40.418696 19321 tablet_bootstrap.cc:654] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:40.410773 20295 raft_consensus.cc:513] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.437779 20295 leader_election.cc:290] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:40.467108 19322 tablet_bootstrap.cc:492] T 42230a0bc94249fda91cf79fba48f024 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:40.467778 19322 ts_tablet_manager.cc:1397] T 42230a0bc94249fda91cf79fba48f024 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.223s user 0.009s sys 0.003s
I20250111 02:57:40.510478 19322 raft_consensus.cc:357] T 42230a0bc94249fda91cf79fba48f024 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.511380 19322 raft_consensus.cc:383] T 42230a0bc94249fda91cf79fba48f024 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:40.511646 19322 raft_consensus.cc:738] T 42230a0bc94249fda91cf79fba48f024 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:40.528360 19330 tablet_bootstrap.cc:492] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:40.528834 19330 ts_tablet_manager.cc:1397] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.289s user 0.013s sys 0.000s
I20250111 02:57:40.532166 19330 raft_consensus.cc:357] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.532946 19330 raft_consensus.cc:383] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:40.534952 20385 raft_consensus.cc:491] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:40.535522 20385 raft_consensus.cc:513] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.537797 20385 leader_election.cc:290] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:40.533216 19330 raft_consensus.cc:738] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:40.538842 19330 consensus_queue.cc:260] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.558879 19330 ts_tablet_manager.cc:1428] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20250111 02:57:40.571857 20296 raft_consensus.cc:491] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:40.572443 20296 raft_consensus.cc:513] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.575012 20296 leader_election.cc:290] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:40.577328 19322 consensus_queue.cc:260] T 42230a0bc94249fda91cf79fba48f024 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.578274 20381 raft_consensus.cc:491] T 42230a0bc94249fda91cf79fba48f024 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:40.578819 20381 raft_consensus.cc:513] T 42230a0bc94249fda91cf79fba48f024 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.579633 19322 ts_tablet_manager.cc:1428] T 42230a0bc94249fda91cf79fba48f024 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.071s user 0.006s sys 0.000s
I20250111 02:57:40.583247 20383 raft_consensus.cc:491] T 2009d6f54e2842e8be3af2cc9147dc2c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:40.583792 20383 raft_consensus.cc:513] T 2009d6f54e2842e8be3af2cc9147dc2c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.586328 20383 leader_election.cc:290] T 2009d6f54e2842e8be3af2cc9147dc2c P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:40.586905 20381 leader_election.cc:290] T 42230a0bc94249fda91cf79fba48f024 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:40.593639 20381 raft_consensus.cc:491] T df1870cceafe463b8d024e8b065850e0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:40.594110 20381 raft_consensus.cc:513] T df1870cceafe463b8d024e8b065850e0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.596683 20381 leader_election.cc:290] T df1870cceafe463b8d024e8b065850e0 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:40.599545 19330 tablet_bootstrap.cc:492] T 64abb30da7fc452cad774bbe59b498f4 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:40.599464 19186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2009d6f54e2842e8be3af2cc9147dc2c" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:40.603384 19187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42230a0bc94249fda91cf79fba48f024" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:40.605494 19185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1870cceafe463b8d024e8b065850e0" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:40.607331 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa86b1e6eca7403c9a9c52592b8fb636" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:40.608060 19193 raft_consensus.cc:3054] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:40.618026 19330 tablet_bootstrap.cc:654] T 64abb30da7fc452cad774bbe59b498f4 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:40.620800 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff06134303d4cb0b4292fe1c0c2d443" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:40.621096 18897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa86b1e6eca7403c9a9c52592b8fb636" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:40.621862 19196 raft_consensus.cc:2463] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:40.628160 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42230a0bc94249fda91cf79fba48f024" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:40.628410 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2009d6f54e2842e8be3af2cc9147dc2c" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:40.629612 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1870cceafe463b8d024e8b065850e0" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:40.633715 18841 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Not found: Tablet not found: 32afc2f158254811836fcf00f7f409ed
I20250111 02:57:40.635829 19322 tablet_bootstrap.cc:492] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:40.643211 18986 leader_election.cc:304] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:40.645503 20229 raft_consensus.cc:2798] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:40.645907 20229 raft_consensus.cc:491] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:40.646265 20229 raft_consensus.cc:3054] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:40.650009 20357 consensus_queue.cc:1035] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:40.650561 18897 raft_consensus.cc:3054] T 42230a0bc94249fda91cf79fba48f024 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:40.651369 20357 consensus_queue.cc:1035] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:40.664847 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff06134303d4cb0b4292fe1c0c2d443" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:40.668141 18903 raft_consensus.cc:3054] T df1870cceafe463b8d024e8b065850e0 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:40.669521 20229 raft_consensus.cc:513] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.677266 19185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff06134303d4cb0b4292fe1c0c2d443" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:40.677958 19185 raft_consensus.cc:3054] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:40.695827 19321 tablet_bootstrap.cc:492] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:40.697630 19321 ts_tablet_manager.cc:1397] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.376s user 0.014s sys 0.009s
W20250111 02:57:40.689095 18842 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: 32afc2f158254811836fcf00f7f409ed
I20250111 02:57:40.710387 18842 leader_election.cc:304] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:40.724788 19321 raft_consensus.cc:357] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.730944 19321 raft_consensus.cc:383] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:40.731325 19321 raft_consensus.cc:738] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:40.731997 19321 consensus_queue.cc:260] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.733932 19321 ts_tablet_manager.cc:1428] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
W20250111 02:57:40.737282 18842 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 48223.9us
I20250111 02:57:40.738318 20381 raft_consensus.cc:2743] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:40.740141 18842 leader_election.cc:304] T 42230a0bc94249fda91cf79fba48f024 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:40.743453 18842 leader_election.cc:304] T 2009d6f54e2842e8be3af2cc9147dc2c P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:40.746628 19322 tablet_bootstrap.cc:654] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:40.747102 20385 raft_consensus.cc:2743] T 2009d6f54e2842e8be3af2cc9147dc2c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:40.748296 18842 leader_election.cc:304] T df1870cceafe463b8d024e8b065850e0 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:40.748878 20151 raft_consensus.cc:491] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:40.749485 20151 raft_consensus.cc:513] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.752769 20151 leader_election.cc:290] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:40.752614 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f71f8dd2894d40b3bfdd20c1453787" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:40.731616 20402 consensus_queue.cc:1035] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20250111 02:57:40.754324 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f71f8dd2894d40b3bfdd20c1453787" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:40.755110 19133 leader_election.cc:343] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:40.765378 19053 raft_consensus.cc:2463] T 36f71f8dd2894d40b3bfdd20c1453787 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:40.762555 19193 raft_consensus.cc:2463] T fa86b1e6eca7403c9a9c52592b8fb636 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:40.766978 19131 leader_election.cc:304] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:40.768193 20151 raft_consensus.cc:2798] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:40.768639 20151 raft_consensus.cc:491] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:40.769287 20151 raft_consensus.cc:3054] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:40.770380 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff06134303d4cb0b4292fe1c0c2d443" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:40.771519 19185 raft_consensus.cc:2463] T 6ff06134303d4cb0b4292fe1c0c2d443 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
W20250111 02:57:40.715339 18989 leader_election.cc:343] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:40.772006 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 56725us
I20250111 02:57:40.772971 20229 leader_election.cc:290] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:40.777278 18986 leader_election.cc:304] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:40.783483 19321 tablet_bootstrap.cc:492] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:40.787482 20310 raft_consensus.cc:491] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:40.787999 20310 raft_consensus.cc:513] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.790237 20310 leader_election.cc:290] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:40.794523 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e4b222bf974a8d80b25141a5686761" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:40.803175 18986 leader_election.cc:304] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:40.803326 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e4b222bf974a8d80b25141a5686761" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:40.803927 20357 raft_consensus.cc:2798] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:40.804868 20357 raft_consensus.cc:695] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:40.805848 20402 raft_consensus.cc:2798] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:40.806802 19321 tablet_bootstrap.cc:654] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:40.806869 20402 raft_consensus.cc:695] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:40.808120 20402 consensus_queue.cc:237] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.813794 20357 consensus_queue.cc:237] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:40.820235 18989 consensus_peers.cc:487] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:40.821653 20401 consensus_queue.cc:1035] T 2873325cece64db6a673d4fe08eb7409 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:40.824641 20310 raft_consensus.cc:491] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:40.825248 20310 raft_consensus.cc:513] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:40.827723 19321 tablet_bootstrap.cc:492] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:40.828145 19321 ts_tablet_manager.cc:1397] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.002s
I20250111 02:57:40.829749 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076aafe3958948079e42501c6bd86153" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:40.831321 20385 raft_consensus.cc:2743] T df1870cceafe463b8d024e8b065850e0 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:40.831449 19133 leader_election.cc:343] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:40.832038 19133 leader_election.cc:343] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:40.829032 20310 leader_election.cc:290] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:40.837275 20381 raft_consensus.cc:2743] T 42230a0bc94249fda91cf79fba48f024 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:40.854010 20151 raft_consensus.cc:513] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:40.855615 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 52559.2us
W20250111 02:57:40.857448 19131 leader_election.cc:343] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:40.858107 19131 leader_election.cc:304] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:40.861449 20404 raft_consensus.cc:2743] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:40.872079 19321 raft_consensus.cc:357] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:40.872864 19321 raft_consensus.cc:383] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:40.873142 19321 raft_consensus.cc:738] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:40.873754 19321 consensus_queue.cc:260] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:40.876763 19330 tablet_bootstrap.cc:492] T 64abb30da7fc452cad774bbe59b498f4 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:40.882287 20151 leader_election.cc:290] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:40.899466 19330 ts_tablet_manager.cc:1397] T 64abb30da7fc452cad774bbe59b498f4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.300s user 0.007s sys 0.005s
I20250111 02:57:40.917516 20419 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:40.923470 18897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f71f8dd2894d40b3bfdd20c1453787" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:40.930497 19133 leader_election.cc:343] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:40.932935 19321 ts_tablet_manager.cc:1428] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.105s user 0.005s sys 0.002s
I20250111 02:57:40.933517 20310 raft_consensus.cc:491] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:40.933962 19321 tablet_bootstrap.cc:492] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:40.934137 20310 raft_consensus.cc:513] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:40.937042 20310 leader_election.cc:290] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:40.895525 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076aafe3958948079e42501c6bd86153" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:40.896365 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f71f8dd2894d40b3bfdd20c1453787" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:40.940598 19051 raft_consensus.cc:3054] T 36f71f8dd2894d40b3bfdd20c1453787 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:40.880357 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 58931.5us
I20250111 02:57:40.944990 19321 tablet_bootstrap.cc:654] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:40.948773 19051 raft_consensus.cc:2463] T 36f71f8dd2894d40b3bfdd20c1453787 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
W20250111 02:57:40.949031 18989 leader_election.cc:343] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:40.958988 18897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64abb30da7fc452cad774bbe59b498f4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:40.962283 19322 tablet_bootstrap.cc:492] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
W20250111 02:57:40.950960 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 55296.4us
I20250111 02:57:40.964646 20419 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:40.973593 20421 tablet.cc:1721] T 7ea78feee64340e39404cba0f1bf04db P 54628fbcec2f4359aa9e26a734d4d29f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250111 02:57:40.990945 19133 leader_election.cc:343] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:40.973593 18989 consensus_peers.cc:487] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:41.002244 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
W20250111 02:57:41.004079 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 47465.1us
I20250111 02:57:41.018761 19322 ts_tablet_manager.cc:1397] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.383s user 0.013s sys 0.003s
W20250111 02:57:41.023998 19131 leader_election.cc:343] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:41.024544 19131 leader_election.cc:304] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:41.028606 19322 raft_consensus.cc:357] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.029901 20419 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:41.036315 20418 raft_consensus.cc:2743] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:41.049284 19131 leader_election.cc:304] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:41.058190 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64abb30da7fc452cad774bbe59b498f4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:40.997813 19330 raft_consensus.cc:357] T 64abb30da7fc452cad774bbe59b498f4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.062558 19330 raft_consensus.cc:383] T 64abb30da7fc452cad774bbe59b498f4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:41.062984 19330 raft_consensus.cc:738] T 64abb30da7fc452cad774bbe59b498f4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:41.063791 19330 consensus_queue.cc:260] T 64abb30da7fc452cad774bbe59b498f4 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.064940 19051 raft_consensus.cc:2463] T 64abb30da7fc452cad774bbe59b498f4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:41.066115 19330 ts_tablet_manager.cc:1428] T 64abb30da7fc452cad774bbe59b498f4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.128s user 0.006s sys 0.000s
I20250111 02:57:41.067766 20425 tablet.cc:1721] T 7ea78feee64340e39404cba0f1bf04db P d0cedab82b154d62b78809edc6f28bfa: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250111 02:57:41.040653 18989 leader_election.cc:343] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:41.095640 20420 raft_consensus.cc:2798] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:41.103477 18897 raft_consensus.cc:3054] T 2009d6f54e2842e8be3af2cc9147dc2c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:41.112018 19322 raft_consensus.cc:383] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:41.113371 19322 raft_consensus.cc:738] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:41.149674 19322 consensus_queue.cc:260] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.180282 19321 tablet_bootstrap.cc:492] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:41.180729 19321 ts_tablet_manager.cc:1397] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.247s user 0.025s sys 0.012s
I20250111 02:57:41.191483 20420 raft_consensus.cc:695] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:41.192483 20420 consensus_queue.cc:237] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.202584 19322 ts_tablet_manager.cc:1428] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.176s user 0.005s sys 0.001s
I20250111 02:57:41.203725 19330 tablet_bootstrap.cc:492] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:41.203747 20427 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:41.211309 20427 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:41.214056 20427 leader_election.cc:290] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8784b9603f8544b79373158aace063af (127.18.20.125:40359), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:41.215131 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:41.228449 20310 raft_consensus.cc:491] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:41.231360 20310 raft_consensus.cc:513] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.230006 19321 raft_consensus.cc:357] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.235206 19321 raft_consensus.cc:383] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:41.235491 19321 raft_consensus.cc:738] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:41.236114 19321 consensus_queue.cc:260] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.237773 19321 ts_tablet_manager.cc:1428] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.057s user 0.004s sys 0.000s
I20250111 02:57:41.252640 20151 raft_consensus.cc:491] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:41.253135 20151 raft_consensus.cc:513] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:41.234897 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d83cfd85a52f438dabe49b6223d5090f" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:41.260035 20151 leader_election.cc:290] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:41.273897 19133 leader_election.cc:343] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:41.277281 20310 leader_election.cc:290] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:41.257899 19322 tablet_bootstrap.cc:492] T 8eb87813c28742edad99470c84269ef4 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:41.271222 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31ad2e1895d4a26b8e9f7cd7366ef81" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:41.286502 19133 leader_election.cc:343] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:41.290654 19131 leader_election.cc:304] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:41.291851 20310 raft_consensus.cc:2798] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:41.298916 20310 raft_consensus.cc:491] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:41.313643 20310 raft_consensus.cc:3054] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:41.323944 20431 tablet.cc:1721] T 7ea78feee64340e39404cba0f1bf04db P 8221604733fe4fc0a1fc32de8d6af71c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250111 02:57:41.333469 19330 tablet_bootstrap.cc:654] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:41.338109 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d83cfd85a52f438dabe49b6223d5090f" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:41.343634 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31ad2e1895d4a26b8e9f7cd7366ef81" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:41.356513 18989 consensus_peers.cc:487] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:41.330696 19322 tablet_bootstrap.cc:654] T 8eb87813c28742edad99470c84269ef4 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:41.398232 20381 raft_consensus.cc:491] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:41.398753 20381 raft_consensus.cc:513] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.400795 20381 leader_election.cc:290] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:41.403043 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6da575f052d4b20a8e41f0bd9dab7e4" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:41.407428 19131 leader_election.cc:343] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:41.407848 19131 leader_election.cc:304] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:41.408514 20420 raft_consensus.cc:2743] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:41.411379 19131 leader_election.cc:343] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:41.411795 19131 leader_election.cc:304] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:41.412585 20420 raft_consensus.cc:2743] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:41.429332 20310 raft_consensus.cc:513] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.432864 20402 consensus_queue.cc:1035] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:41.476874 20310 leader_election.cc:290] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:41.477767 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64abb30da7fc452cad774bbe59b498f4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:41.479274 19133 leader_election.cc:343] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:41.482940 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6da575f052d4b20a8e41f0bd9dab7e4" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:41.487197 18842 leader_election.cc:304] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:41.488113 20381 raft_consensus.cc:2743] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:41.538424 19330 tablet_bootstrap.cc:492] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:41.538884 19330 ts_tablet_manager.cc:1397] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.335s user 0.004s sys 0.015s
W20250111 02:57:41.536006 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 48659.2us
I20250111 02:57:41.541221 19330 raft_consensus.cc:357] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.541954 19330 raft_consensus.cc:383] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:41.542204 19330 raft_consensus.cc:738] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
W20250111 02:57:41.545320 18989 consensus_peers.cc:487] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:41.575920 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64abb30da7fc452cad774bbe59b498f4" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:41.576594 19051 raft_consensus.cc:3054] T 64abb30da7fc452cad774bbe59b498f4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:41.588024 19051 raft_consensus.cc:2463] T 64abb30da7fc452cad774bbe59b498f4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:41.592295 19322 tablet_bootstrap.cc:492] T 8eb87813c28742edad99470c84269ef4 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:41.592746 19322 ts_tablet_manager.cc:1397] T 8eb87813c28742edad99470c84269ef4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.335s user 0.015s sys 0.003s
I20250111 02:57:41.615370 20401 raft_consensus.cc:491] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0cedab82b154d62b78809edc6f28bfa)
I20250111 02:57:41.616015 20401 raft_consensus.cc:513] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.617980 20401 leader_election.cc:290] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:41.626220 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 80388.5us
I20250111 02:57:41.633551 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:41.634431 18903 raft_consensus.cc:2388] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54628fbcec2f4359aa9e26a734d4d29f in current term 2: Already voted for candidate 8221604733fe4fc0a1fc32de8d6af71c in this term.
I20250111 02:57:41.542801 19330 consensus_queue.cc:260] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.648139 19330 ts_tablet_manager.cc:1428] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.109s user 0.005s sys 0.000s
I20250111 02:57:41.649101 19330 tablet_bootstrap.cc:492] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:41.656088 19322 raft_consensus.cc:357] T 8eb87813c28742edad99470c84269ef4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.667645 19322 raft_consensus.cc:383] T 8eb87813c28742edad99470c84269ef4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:41.667960 19322 raft_consensus.cc:738] T 8eb87813c28742edad99470c84269ef4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:41.668123 18903 raft_consensus.cc:3054] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:41.668593 19322 consensus_queue.cc:260] T 8eb87813c28742edad99470c84269ef4 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.684386 19131 leader_election.cc:304] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:41.686571 20310 raft_consensus.cc:2798] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:41.687093 20310 raft_consensus.cc:695] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:41.687891 20310 consensus_queue.cc:237] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.706775 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:41.711898 19194 raft_consensus.cc:2463] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 2.
I20250111 02:57:41.670888 19322 ts_tablet_manager.cc:1428] T 8eb87813c28742edad99470c84269ef4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.078s user 0.007s sys 0.000s
I20250111 02:57:41.732803 19322 tablet_bootstrap.cc:492] T 36f71f8dd2894d40b3bfdd20c1453787 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:41.747653 19330 tablet_bootstrap.cc:654] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:41.792609 19322 tablet_bootstrap.cc:654] T 36f71f8dd2894d40b3bfdd20c1453787 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:41.845721 18986 leader_election.cc:304] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:41.846599 20402 raft_consensus.cc:2798] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 02:57:41.847018 20402 raft_consensus.cc:491] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0cedab82b154d62b78809edc6f28bfa)
I20250111 02:57:41.847505 20402 raft_consensus.cc:3054] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:57:41.854676 19330 tablet_bootstrap.cc:492] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:41.855229 19330 ts_tablet_manager.cc:1397] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.206s user 0.013s sys 0.011s
I20250111 02:57:41.856832 20402 raft_consensus.cc:513] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.857836 19330 raft_consensus.cc:357] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.871531 19330 raft_consensus.cc:383] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:41.871887 19330 raft_consensus.cc:738] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:41.872576 19330 consensus_queue.cc:260] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.874572 18899 raft_consensus.cc:3054] T 8eb87813c28742edad99470c84269ef4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:41.893402 19322 tablet_bootstrap.cc:492] T 36f71f8dd2894d40b3bfdd20c1453787 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:41.898639 19322 ts_tablet_manager.cc:1397] T 36f71f8dd2894d40b3bfdd20c1453787 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.165s user 0.009s sys 0.003s
I20250111 02:57:41.889971 20441 consensus_queue.cc:1035] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:41.892081 20401 consensus_queue.cc:1035] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:41.907685 19330 ts_tablet_manager.cc:1428] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.052s user 0.005s sys 0.001s
I20250111 02:57:41.909160 20402 leader_election.cc:290] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 2 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:41.931314 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 50849.4us
I20250111 02:57:41.941147 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:41.941937 18903 raft_consensus.cc:2388] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 54628fbcec2f4359aa9e26a734d4d29f in current term 2: Already voted for candidate 8221604733fe4fc0a1fc32de8d6af71c in this term.
I20250111 02:57:41.953931 19330 tablet_bootstrap.cc:492] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:41.957643 20310 raft_consensus.cc:491] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:41.958135 20310 raft_consensus.cc:513] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.960402 20310 leader_election.cc:290] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:41.962050 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f4500d3beb4c9bb20f84c661c3758b" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:41.963897 19133 leader_election.cc:343] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:41.917779 19322 raft_consensus.cc:357] T 36f71f8dd2894d40b3bfdd20c1453787 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.968106 19322 raft_consensus.cc:383] T 36f71f8dd2894d40b3bfdd20c1453787 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:41.968999 19322 raft_consensus.cc:738] T 36f71f8dd2894d40b3bfdd20c1453787 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:41.970388 19322 consensus_queue.cc:260] T 36f71f8dd2894d40b3bfdd20c1453787 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:41.981439 18989 consensus_peers.cc:487] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:41.985740 20310 raft_consensus.cc:491] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:41.985916 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:41.986325 20310 raft_consensus.cc:513] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:41.988603 20310 leader_election.cc:290] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:41.990411 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c669a3ff8aa46fa8b04b86ed3bfde49" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:41.992008 19133 leader_election.cc:343] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:42.020807 19322 ts_tablet_manager.cc:1428] T 36f71f8dd2894d40b3bfdd20c1453787 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.120s user 0.007s sys 0.001s
I20250111 02:57:42.022964 20401 raft_consensus.cc:491] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250111 02:57:42.029918 18614 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 3514d795a0f24adcaa2accc0cf05b142 (table default.Table_4c9aebb06d554107a5448d968fdb0436 [id=3fab00618e6548de85b5d20adee4e9ac], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3514d795a0f24adcaa2accc0cf05b142 consensus state has no leader
I20250111 02:57:42.023538 20401 raft_consensus.cc:513] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:42.039330 20401 leader_election.cc:290] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:42.042249 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f4500d3beb4c9bb20f84c661c3758b" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:42.054463 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c669a3ff8aa46fa8b04b86ed3bfde49" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:42.063252 19194 raft_consensus.cc:2463] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 2.
I20250111 02:57:42.075914 19330 tablet_bootstrap.cc:654] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:42.079985 19322 tablet_bootstrap.cc:492] T e01771c56ea5498a876d3bb865c752a2 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:42.081802 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f561cfd4f34426b7e42027e53b3ff1" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:42.086272 19131 leader_election.cc:343] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:42.090335 19131 leader_election.cc:304] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:42.091223 20310 raft_consensus.cc:2743] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:57:42.092500 19131 leader_election.cc:343] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:42.092862 19131 leader_election.cc:304] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:42.093910 20310 raft_consensus.cc:2743] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:42.094782 19195 raft_consensus.cc:2463] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
W20250111 02:57:42.119549 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 45782.9us
W20250111 02:57:42.162828 18693 leader_election.cc:336] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.553s (SENT)
I20250111 02:57:42.163363 18693 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
W20250111 02:57:42.177446 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 54205.4us
I20250111 02:57:42.182595 18986 leader_election.cc:304] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:42.196210 20452 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250111 02:57:42.200862 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f561cfd4f34426b7e42027e53b3ff1" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:42.224390 20441 raft_consensus.cc:2798] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:42.224813 20441 raft_consensus.cc:491] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:42.225157 20441 raft_consensus.cc:3054] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:42.231202 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 48695.1us
I20250111 02:57:42.232035 18986 leader_election.cc:304] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:42.233250 20229 raft_consensus.cc:2798] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:57:42.233700 20229 raft_consensus.cc:695] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 2 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:42.234534 20229 consensus_queue.cc:237] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:42.245427 20441 raft_consensus.cc:513] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:42.265774 18989 leader_election.cc:343] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:42.274658 19330 tablet_bootstrap.cc:492] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:42.295495 19322 tablet_bootstrap.cc:654] T e01771c56ea5498a876d3bb865c752a2 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:42.317782 19330 ts_tablet_manager.cc:1397] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.364s user 0.007s sys 0.007s
I20250111 02:57:42.321206 20381 raft_consensus.cc:491] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:42.321697 20381 raft_consensus.cc:513] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:42.324883 20381 leader_election.cc:290] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:42.326992 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:42.327617 19195 raft_consensus.cc:2463] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 2.
I20250111 02:57:42.328678 18841 leader_election.cc:304] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:42.329598 20381 raft_consensus.cc:2798] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250111 02:57:42.329965 20381 raft_consensus.cc:491] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:42.330344 20381 raft_consensus.cc:3054] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:57:42.278183 20441 leader_election.cc:290] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:42.356781 20381 raft_consensus.cc:513] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:42.366564 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f561cfd4f34426b7e42027e53b3ff1" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:42.367400 19194 raft_consensus.cc:3054] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:42.359783 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f561cfd4f34426b7e42027e53b3ff1" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:42.395665 20381 leader_election.cc:290] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 3 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:42.397178 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:42.397816 19195 raft_consensus.cc:3054] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 2 FOLLOWER]: Advancing to term 3
W20250111 02:57:42.402863 19594 leader_election.cc:336] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.588s (SENT)
I20250111 02:57:42.403406 19594 leader_election.cc:304] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727; no voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c
W20250111 02:57:42.415551 18989 consensus_peers.cc:487] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:42.415616 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:42.415700 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef92ac716824107916a1ea4fd49d5aa" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:42.320623 19330 raft_consensus.cc:357] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:42.419970 19330 raft_consensus.cc:383] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:42.420282 19330 raft_consensus.cc:738] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:42.421047 19330 consensus_queue.cc:260] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:42.438428 19322 tablet_bootstrap.cc:492] T e01771c56ea5498a876d3bb865c752a2 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:42.438887 19322 ts_tablet_manager.cc:1397] T e01771c56ea5498a876d3bb865c752a2 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.359s user 0.014s sys 0.000s
I20250111 02:57:42.438988 20458 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250111 02:57:42.441397 19322 raft_consensus.cc:357] T e01771c56ea5498a876d3bb865c752a2 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:42.442162 19322 raft_consensus.cc:383] T e01771c56ea5498a876d3bb865c752a2 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:42.442438 19322 raft_consensus.cc:738] T e01771c56ea5498a876d3bb865c752a2 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:42.445554 18899 raft_consensus.cc:3054] T 36f71f8dd2894d40b3bfdd20c1453787 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:42.454406 19133 consensus_peers.cc:487] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:42.455451 19594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 52637.1us
I20250111 02:57:42.458871 20310 consensus_queue.cc:1035] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:42.460434 19330 ts_tablet_manager.cc:1428] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.142s user 0.006s sys 0.000s
I20250111 02:57:42.464903 19195 raft_consensus.cc:2463] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 3.
I20250111 02:57:42.466076 18841 leader_election.cc:304] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:42.466931 20456 raft_consensus.cc:2798] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 3 FOLLOWER]: Leader election won for term 3
I20250111 02:57:42.467427 20456 raft_consensus.cc:695] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 3 LEADER]: Becoming Leader. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Running, Role: LEADER
I20250111 02:57:42.468230 20456 consensus_queue.cc:237] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:42.473399 20310 consensus_queue.cc:1035] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250111 02:57:42.476353 18989 leader_election.cc:343] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:42.477700 19322 consensus_queue.cc:260] T e01771c56ea5498a876d3bb865c752a2 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:42.503209 19194 raft_consensus.cc:2463] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:42.533553 19322 ts_tablet_manager.cc:1428] T e01771c56ea5498a876d3bb865c752a2 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.094s user 0.005s sys 0.000s
I20250111 02:57:42.545691 19322 tablet_bootstrap.cc:492] T aca35147034446b3882b5244923bddc7 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
W20250111 02:57:42.549611 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 53879.1us
I20250111 02:57:42.597353 19330 tablet_bootstrap.cc:492] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:42.618638 19322 tablet_bootstrap.cc:654] T aca35147034446b3882b5244923bddc7 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:42.639570 18986 leader_election.cc:304] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:42.641090 20401 raft_consensus.cc:2798] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:42.641634 20401 raft_consensus.cc:695] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:42.642642 20401 consensus_queue.cc:237] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:42.653046 19330 tablet_bootstrap.cc:654] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:42.708206 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 46951.9us
I20250111 02:57:42.722887 20463 raft_consensus.cc:491] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:42.723560 20463 raft_consensus.cc:513] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:42.725711 19322 tablet_bootstrap.cc:492] T aca35147034446b3882b5244923bddc7 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:42.726172 19322 ts_tablet_manager.cc:1397] T aca35147034446b3882b5244923bddc7 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.181s user 0.010s sys 0.000s
I20250111 02:57:42.728832 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b7616fe56f4e5fb49bdde6961fa008" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:42.729575 19322 raft_consensus.cc:357] T aca35147034446b3882b5244923bddc7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:42.730279 19322 raft_consensus.cc:383] T aca35147034446b3882b5244923bddc7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:42.730535 19322 raft_consensus.cc:738] T aca35147034446b3882b5244923bddc7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
W20250111 02:57:42.730654 19133 leader_election.cc:343] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:42.731154 19322 consensus_queue.cc:260] T aca35147034446b3882b5244923bddc7 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:42.732554 20463 leader_election.cc:290] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:42.733510 19322 ts_tablet_manager.cc:1428] T aca35147034446b3882b5244923bddc7 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250111 02:57:42.758200 18899 raft_consensus.cc:3054] T aca35147034446b3882b5244923bddc7 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:42.782433 20401 raft_consensus.cc:491] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:42.783298 20401 raft_consensus.cc:513] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:42.786954 20401 leader_election.cc:290] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:42.800238 19330 tablet_bootstrap.cc:492] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:42.802877 19322 tablet_bootstrap.cc:492] T fa86b1e6eca7403c9a9c52592b8fb636 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:42.803295 19330 ts_tablet_manager.cc:1397] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.204s user 0.012s sys 0.000s
I20250111 02:57:42.812628 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d83cfd85a52f438dabe49b6223d5090f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:42.817914 19052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b7616fe56f4e5fb49bdde6961fa008" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:42.819898 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d83cfd85a52f438dabe49b6223d5090f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:42.820652 19194 raft_consensus.cc:2463] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:42.848618 19330 raft_consensus.cc:357] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:42.849653 19330 raft_consensus.cc:383] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:42.850277 19330 raft_consensus.cc:738] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
W20250111 02:57:42.857412 18989 leader_election.cc:343] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:42.859637 18903 raft_consensus.cc:3054] T e01771c56ea5498a876d3bb865c752a2 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:42.889510 19322 tablet_bootstrap.cc:654] T fa86b1e6eca7403c9a9c52592b8fb636 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:42.907052 20473 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:42.907771 20473 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:42.910285 20473 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:42.916054 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
W20250111 02:57:42.921569 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 55129.6us
I20250111 02:57:42.851773 19330 consensus_queue.cc:260] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:42.930336 19330 ts_tablet_manager.cc:1428] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.127s user 0.006s sys 0.000s
I20250111 02:57:42.931385 19330 tablet_bootstrap.cc:492] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:42.939205 18986 leader_election.cc:304] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:42.942492 20229 raft_consensus.cc:2798] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:42.943367 20229 raft_consensus.cc:491] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:42.943804 20229 raft_consensus.cc:3054] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:42.947280 20463 raft_consensus.cc:491] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:42.947902 20463 raft_consensus.cc:513] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:42.951258 20463 leader_election.cc:290] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:42.952431 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaab428f798a44f69c80e5792d56b920" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:42.954030 19133 leader_election.cc:343] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:42.958514 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 69713.4us
I20250111 02:57:42.970855 20461 raft_consensus.cc:491] T aca35147034446b3882b5244923bddc7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 54628fbcec2f4359aa9e26a734d4d29f)
I20250111 02:57:42.971516 20461 raft_consensus.cc:513] T aca35147034446b3882b5244923bddc7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:42.973995 20461 leader_election.cc:290] T aca35147034446b3882b5244923bddc7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:42.974874 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca35147034446b3882b5244923bddc7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:42.984009 19131 leader_election.cc:343] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:42.983893 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca35147034446b3882b5244923bddc7" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:42.984436 19131 leader_election.cc:304] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:42.986960 20462 raft_consensus.cc:2743] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:42.987715 20462 consensus_queue.cc:1035] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:43.013640 19322 tablet_bootstrap.cc:492] T fa86b1e6eca7403c9a9c52592b8fb636 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:43.014104 19322 ts_tablet_manager.cc:1397] T fa86b1e6eca7403c9a9c52592b8fb636 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.211s user 0.010s sys 0.000s
I20250111 02:57:43.025875 18842 leader_election.cc:304] T aca35147034446b3882b5244923bddc7 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:43.027045 20461 raft_consensus.cc:2743] T aca35147034446b3882b5244923bddc7 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:43.036226 18899 raft_consensus.cc:1235] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 3 LEADER]: Rejecting Update request from peer 54628fbcec2f4359aa9e26a734d4d29f for earlier term 2. Current term is 3. Ops: []
I20250111 02:57:43.042541 19322 raft_consensus.cc:357] T fa86b1e6eca7403c9a9c52592b8fb636 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.043313 19322 raft_consensus.cc:383] T fa86b1e6eca7403c9a9c52592b8fb636 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:43.043571 19322 raft_consensus.cc:738] T fa86b1e6eca7403c9a9c52592b8fb636 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:43.044207 19322 consensus_queue.cc:260] T fa86b1e6eca7403c9a9c52592b8fb636 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.046447 19322 ts_tablet_manager.cc:1428] T fa86b1e6eca7403c9a9c52592b8fb636 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20250111 02:57:43.085115 20296 consensus_queue.cc:1046] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250111 02:57:43.086422 20296 consensus_queue.cc:1225] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Peer 8221604733fe4fc0a1fc32de8d6af71c log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250111 02:57:43.086994 20441 raft_consensus.cc:3049] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 2 LEADER]: Stepping down as leader of term 2
I20250111 02:57:43.087399 20441 raft_consensus.cc:738] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:43.088254 20441 consensus_queue.cc:260] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.096567 20229 raft_consensus.cc:513] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.099937 20441 raft_consensus.cc:3054] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:57:43.112525 20296 raft_consensus.cc:491] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:43.113202 20296 raft_consensus.cc:513] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:43.115545 20296 leader_election.cc:290] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:43.119879 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaab428f798a44f69c80e5792d56b920" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:43.121941 19322 tablet_bootstrap.cc:492] T b2f561cfd4f34426b7e42027e53b3ff1 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:43.142524 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f15096ea484b499fbdecec74eb6f66" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:43.147559 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d83cfd85a52f438dabe49b6223d5090f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:43.138391 20229 leader_election.cc:290] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:43.139281 19330 tablet_bootstrap.cc:654] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:43.174413 20296 raft_consensus.cc:491] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:43.175026 20296 raft_consensus.cc:513] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:43.183921 18989 leader_election.cc:343] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:43.188859 20480 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:43.191350 20480 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:43.195588 19322 tablet_bootstrap.cc:654] T b2f561cfd4f34426b7e42027e53b3ff1 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:43.201417 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
W20250111 02:57:43.203948 18989 leader_election.cc:343] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:43.186859 20296 leader_election.cc:290] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:43.209802 20480 leader_election.cc:290] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8784b9603f8544b79373158aace063af (127.18.20.125:40359), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:43.216926 19330 tablet_bootstrap.cc:492] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:43.217422 19330 ts_tablet_manager.cc:1397] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.286s user 0.009s sys 0.002s
I20250111 02:57:43.227178 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dd0bccf6854d37a7ef95b5ab06308e" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:43.236152 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f15096ea484b499fbdecec74eb6f66" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:43.237123 19194 raft_consensus.cc:2463] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
W20250111 02:57:43.265511 19131 leader_election.cc:343] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:43.266021 19131 leader_election.cc:304] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:43.267287 20462 raft_consensus.cc:2743] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:43.269307 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d83cfd85a52f438dabe49b6223d5090f" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:43.270059 19194 raft_consensus.cc:3054] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:43.293100 19322 tablet_bootstrap.cc:492] T b2f561cfd4f34426b7e42027e53b3ff1 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:43.293574 19322 ts_tablet_manager.cc:1397] T b2f561cfd4f34426b7e42027e53b3ff1 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.172s user 0.010s sys 0.000s
I20250111 02:57:43.295567 19330 raft_consensus.cc:357] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.297462 19330 raft_consensus.cc:383] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:43.300488 19322 raft_consensus.cc:357] T b2f561cfd4f34426b7e42027e53b3ff1 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.301210 19322 raft_consensus.cc:383] T b2f561cfd4f34426b7e42027e53b3ff1 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:43.301468 19322 raft_consensus.cc:738] T b2f561cfd4f34426b7e42027e53b3ff1 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:43.302096 19322 consensus_queue.cc:260] T b2f561cfd4f34426b7e42027e53b3ff1 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.305868 19194 raft_consensus.cc:2463] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
W20250111 02:57:43.306452 18989 leader_election.cc:343] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:43.320722 18986 consensus_peers.cc:487] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cf1ce65ccf54358b95665b9764fbdb0. This is attempt 6: this message will repeat every 5th retry.
I20250111 02:57:43.299690 19330 raft_consensus.cc:738] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:43.348829 19330 consensus_queue.cc:260] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.359176 19322 ts_tablet_manager.cc:1428] T b2f561cfd4f34426b7e42027e53b3ff1 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.065s user 0.008s sys 0.000s
I20250111 02:57:43.359895 18986 leader_election.cc:304] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:43.361842 20229 raft_consensus.cc:2798] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:43.362236 20229 raft_consensus.cc:491] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:43.362560 20229 raft_consensus.cc:3054] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:43.367013 19193 raft_consensus.cc:1270] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 3 FOLLOWER]: Refusing update from remote peer 8221604733fe4fc0a1fc32de8d6af71c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250111 02:57:43.378387 20461 consensus_queue.cc:1035] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:57:43.380300 20229 raft_consensus.cc:513] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:43.383976 20229 leader_election.cc:290] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:43.387145 19322 tablet_bootstrap.cc:492] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:43.398797 20462 raft_consensus.cc:491] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:43.399358 20462 raft_consensus.cc:513] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.401376 20462 leader_election.cc:290] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:43.403460 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3514d795a0f24adcaa2accc0cf05b142" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:43.404793 19133 leader_election.cc:343] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:43.406154 20461 raft_consensus.cc:491] T b2f561cfd4f34426b7e42027e53b3ff1 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:43.406666 20461 raft_consensus.cc:513] T b2f561cfd4f34426b7e42027e53b3ff1 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.408866 20461 leader_election.cc:290] T b2f561cfd4f34426b7e42027e53b3ff1 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:43.409741 19185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f561cfd4f34426b7e42027e53b3ff1" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:43.410501 19185 raft_consensus.cc:2388] T b2f561cfd4f34426b7e42027e53b3ff1 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:43.412468 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dd0bccf6854d37a7ef95b5ab06308e" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:43.413168 19194 raft_consensus.cc:2463] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:43.420207 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f561cfd4f34426b7e42027e53b3ff1" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:43.423049 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f15096ea484b499fbdecec74eb6f66" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:43.424752 19053 raft_consensus.cc:1270] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 3 FOLLOWER]: Refusing update from remote peer 8221604733fe4fc0a1fc32de8d6af71c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250111 02:57:43.425156 19053 pending_rounds.cc:77] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f: Aborting all ops after (but not including) 1
I20250111 02:57:43.425382 19053 pending_rounds.cc:99] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250111 02:57:43.425624 19053 raft_consensus.cc:2883] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250111 02:57:43.447686 19330 ts_tablet_manager.cc:1428] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.230s user 0.010s sys 0.000s
I20250111 02:57:43.451639 19330 tablet_bootstrap.cc:492] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:43.478974 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f15096ea484b499fbdecec74eb6f66" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:43.480880 19194 raft_consensus.cc:3054] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:43.495637 18842 leader_election.cc:304] T b2f561cfd4f34426b7e42027e53b3ff1 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:43.496296 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3514d795a0f24adcaa2accc0cf05b142" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:43.496620 20381 raft_consensus.cc:3054] T b2f561cfd4f34426b7e42027e53b3ff1 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:43.507428 18989 leader_election.cc:343] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:43.513772 20381 raft_consensus.cc:2743] T b2f561cfd4f34426b7e42027e53b3ff1 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:43.515542 19322 tablet_bootstrap.cc:654] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:43.541707 20441 raft_consensus.cc:491] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:43.542277 20441 raft_consensus.cc:513] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.543824 20462 raft_consensus.cc:491] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:43.544386 20462 raft_consensus.cc:513] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:43.545624 20441 leader_election.cc:290] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:43.547369 20462 leader_election.cc:290] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:43.549152 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c57d85390324bbc9ea30a709faec01c" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:43.551288 19133 leader_election.cc:343] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:43.552083 19330 tablet_bootstrap.cc:654] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:43.553813 18986 leader_election.cc:304] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
W20250111 02:57:43.573774 18693 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.661s (SENT) [suppressed 2 similar messages]
I20250111 02:57:43.583242 20441 raft_consensus.cc:2798] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:43.584154 20441 raft_consensus.cc:695] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:43.585234 20441 consensus_queue.cc:237] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:43.570596 18842 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 73568.4us
I20250111 02:57:43.596360 20461 consensus_queue.cc:1035] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:57:43.587973 18986 leader_election.cc:304] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:43.630511 19330 tablet_bootstrap.cc:492] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:43.631039 19330 ts_tablet_manager.cc:1397] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.180s user 0.010s sys 0.003s
I20250111 02:57:43.633433 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e4b222bf974a8d80b25141a5686761" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:43.633678 19330 raft_consensus.cc:357] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.634435 19330 raft_consensus.cc:383] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:43.634719 19330 raft_consensus.cc:738] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:43.635294 20487 raft_consensus.cc:2798] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:43.635748 20487 raft_consensus.cc:491] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:43.636076 20487 raft_consensus.cc:3054] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:43.644630 19322 tablet_bootstrap.cc:492] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:43.645114 19322 ts_tablet_manager.cc:1397] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.258s user 0.010s sys 0.000s
I20250111 02:57:43.651446 19330 consensus_queue.cc:260] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:43.667492 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 80702us
I20250111 02:57:43.675486 19330 ts_tablet_manager.cc:1428] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.044s user 0.004s sys 0.003s
I20250111 02:57:43.676559 19330 tablet_bootstrap.cc:492] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:43.676506 20461 raft_consensus.cc:491] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 54628fbcec2f4359aa9e26a734d4d29f)
I20250111 02:57:43.677057 20461 raft_consensus.cc:513] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.679111 20461 leader_election.cc:290] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:43.680997 18841 leader_election.cc:343] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Not found: Tablet not found: 4b302fcba27244c193586bc25fc9305f
I20250111 02:57:43.682822 19330 tablet_bootstrap.cc:654] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:43.685927 18989 leader_election.cc:343] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:43.702181 19194 raft_consensus.cc:2463] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:43.707585 19322 raft_consensus.cc:357] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.708370 19322 raft_consensus.cc:383] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:43.708660 19322 raft_consensus.cc:738] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:43.709314 19322 consensus_queue.cc:260] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.729768 19330 tablet_bootstrap.cc:492] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:43.729614 20487 raft_consensus.cc:513] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:43.730269 19330 ts_tablet_manager.cc:1397] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.004s
I20250111 02:57:43.733525 20487 leader_election.cc:290] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:43.752382 19330 raft_consensus.cc:357] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:43.753156 19330 raft_consensus.cc:383] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:43.753535 19330 raft_consensus.cc:738] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:43.754240 19330 consensus_queue.cc:260] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:43.756942 19330 ts_tablet_manager.cc:1428] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.026s user 0.001s sys 0.004s
I20250111 02:57:43.764043 19322 ts_tablet_manager.cc:1428] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.119s user 0.008s sys 0.000s
I20250111 02:57:43.765067 19322 tablet_bootstrap.cc:492] T 64abb30da7fc452cad774bbe59b498f4 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
W20250111 02:57:43.767210 18614 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 845da3ee0863430ea3fe9de8c84f2c17 (table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 845da3ee0863430ea3fe9de8c84f2c17 consensus state has no leader
I20250111 02:57:43.785311 18986 leader_election.cc:304] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:43.786304 20488 raft_consensus.cc:2798] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:43.788375 20488 raft_consensus.cc:695] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:43.789533 20488 consensus_queue.cc:237] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:43.794164 18842 leader_election.cc:343] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: 4b302fcba27244c193586bc25fc9305f
I20250111 02:57:43.794601 18842 leader_election.cc:304] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:43.795248 20461 raft_consensus.cc:2743] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:57:43.800802 20461 raft_consensus.cc:491] T fa86b1e6eca7403c9a9c52592b8fb636 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:43.801293 20461 raft_consensus.cc:513] T fa86b1e6eca7403c9a9c52592b8fb636 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:43.803391 20461 leader_election.cc:290] T fa86b1e6eca7403c9a9c52592b8fb636 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:43.804208 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa86b1e6eca7403c9a9c52592b8fb636" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:43.825897 19131 leader_election.cc:343] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:43.826336 19131 leader_election.cc:304] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:43.827555 20462 raft_consensus.cc:2743] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:43.827929 19185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e4b222bf974a8d80b25141a5686761" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:43.828593 19185 raft_consensus.cc:2463] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:43.831879 19330 tablet_bootstrap.cc:492] T 076aafe3958948079e42501c6bd86153 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:43.841810 19330 tablet_bootstrap.cc:654] T 076aafe3958948079e42501c6bd86153 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:43.848001 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dd0bccf6854d37a7ef95b5ab06308e" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:43.851670 19594 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.653s (SENT) [suppressed 2 similar messages]
I20250111 02:57:43.853346 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa86b1e6eca7403c9a9c52592b8fb636" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:43.857674 20463 raft_consensus.cc:491] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:43.858239 20463 raft_consensus.cc:513] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:43.860327 20463 leader_election.cc:290] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:43.867575 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56fb0bf949904208af003f08caae7594" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:43.869184 19133 leader_election.cc:343] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:43.869603 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dd0bccf6854d37a7ef95b5ab06308e" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:43.870168 19195 raft_consensus.cc:3054] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:43.871950 19322 tablet_bootstrap.cc:654] T 64abb30da7fc452cad774bbe59b498f4 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:43.883721 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c57d85390324bbc9ea30a709faec01c" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:43.888188 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56fb0bf949904208af003f08caae7594" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:43.900220 18842 leader_election.cc:304] T fa86b1e6eca7403c9a9c52592b8fb636 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:43.906423 19195 raft_consensus.cc:2463] T 48dd0bccf6854d37a7ef95b5ab06308e P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:43.902696 20486 raft_consensus.cc:2743] T fa86b1e6eca7403c9a9c52592b8fb636 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:43.911387 20402 raft_consensus.cc:491] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:43.914676 20402 raft_consensus.cc:513] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:43.916829 18989 leader_election.cc:343] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:43.917301 20402 leader_election.cc:290] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:43.921614 20402 consensus_queue.cc:1035] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250111 02:57:43.936681 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 53764.4us
W20250111 02:57:43.959017 18963 tablet.cc:2367] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:57:43.961671 18986 leader_election.cc:304] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:43.962543 20296 raft_consensus.cc:2798] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:43.962978 20296 raft_consensus.cc:491] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:43.963407 20296 raft_consensus.cc:3054] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:43.976987 19322 tablet_bootstrap.cc:492] T 64abb30da7fc452cad774bbe59b498f4 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:43.977478 19322 ts_tablet_manager.cc:1397] T 64abb30da7fc452cad774bbe59b498f4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.213s user 0.007s sys 0.002s
I20250111 02:57:43.980556 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c669a3ff8aa46fa8b04b86ed3bfde49" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:43.980849 18897 raft_consensus.cc:3054] T fa86b1e6eca7403c9a9c52592b8fb636 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:44.010831 18989 leader_election.cc:343] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:44.021240 19322 raft_consensus.cc:357] T 64abb30da7fc452cad774bbe59b498f4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.022055 19322 raft_consensus.cc:383] T 64abb30da7fc452cad774bbe59b498f4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:44.022328 19322 raft_consensus.cc:738] T 64abb30da7fc452cad774bbe59b498f4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:44.048717 19330 tablet_bootstrap.cc:492] T 076aafe3958948079e42501c6bd86153 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:44.049211 19330 ts_tablet_manager.cc:1397] T 076aafe3958948079e42501c6bd86153 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.218s user 0.006s sys 0.007s
I20250111 02:57:44.063405 20487 consensus_queue.cc:1035] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:44.067109 20296 raft_consensus.cc:513] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.070262 20296 leader_election.cc:290] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:44.072247 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e4b222bf974a8d80b25141a5686761" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:44.076447 19330 raft_consensus.cc:357] T 076aafe3958948079e42501c6bd86153 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:44.077494 19330 raft_consensus.cc:383] T 076aafe3958948079e42501c6bd86153 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:44.078045 19330 raft_consensus.cc:738] T 076aafe3958948079e42501c6bd86153 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:44.078847 19330 consensus_queue.cc:260] T 076aafe3958948079e42501c6bd86153 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:44.080546 20463 raft_consensus.cc:491] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:44.080986 20463 raft_consensus.cc:513] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:44.083806 20463 leader_election.cc:290] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:44.092932 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076aafe3958948079e42501c6bd86153" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:44.094362 19133 leader_election.cc:343] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:44.022935 19322 consensus_queue.cc:260] T 64abb30da7fc452cad774bbe59b498f4 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.097014 19322 ts_tablet_manager.cc:1428] T 64abb30da7fc452cad774bbe59b498f4 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.119s user 0.009s sys 0.000s
I20250111 02:57:44.098064 19322 tablet_bootstrap.cc:492] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
W20250111 02:57:44.080121 18989 leader_election.cc:343] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:57:44.138228 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 58155.4us
I20250111 02:57:44.141088 19330 ts_tablet_manager.cc:1428] T 076aafe3958948079e42501c6bd86153 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.091s user 0.005s sys 0.000s
I20250111 02:57:44.093941 18986 leader_election.cc:304] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:44.145058 20487 raft_consensus.cc:2798] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:44.146106 20487 raft_consensus.cc:695] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:44.147049 20487 consensus_queue.cc:237] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:44.150681 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 56849.5us
W20250111 02:57:44.156073 19131 leader_election.cc:343] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:44.156508 19131 leader_election.cc:304] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:44.157168 20463 raft_consensus.cc:2743] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:44.162250 19189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c669a3ff8aa46fa8b04b86ed3bfde49" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:44.162937 19189 raft_consensus.cc:2463] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:44.165530 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e4b222bf974a8d80b25141a5686761" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:44.166131 19197 raft_consensus.cc:3054] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:44.142673 19330 tablet_bootstrap.cc:492] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
W20250111 02:57:44.197281 19131 leader_election.cc:343] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:44.197835 19131 leader_election.cc:304] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0cedab82b154d62b78809edc6f28bfa; no voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:44.199676 20463 raft_consensus.cc:2743] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:44.198832 19197 raft_consensus.cc:2463] T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:44.202914 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076aafe3958948079e42501c6bd86153" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:44.203639 19049 raft_consensus.cc:2463] T 076aafe3958948079e42501c6bd86153 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 0.
I20250111 02:57:44.222265 19322 tablet_bootstrap.cc:654] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:44.233959 18986 leader_election.cc:304] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:44.234786 20296 raft_consensus.cc:2798] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:44.235230 20296 raft_consensus.cc:491] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:44.235584 20296 raft_consensus.cc:3054] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:44.243834 19330 tablet_bootstrap.cc:654] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:44.248879 18693 leader_election.cc:336] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.661s (SENT)
I20250111 02:57:44.249362 18693 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:44.250500 20461 raft_consensus.cc:491] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:44.251003 20461 raft_consensus.cc:513] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.254184 20461 leader_election.cc:290] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:44.255831 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d83cfd85a52f438dabe49b6223d5090f" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:44.255142 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d83cfd85a52f438dabe49b6223d5090f" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:44.257746 19192 raft_consensus.cc:2388] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:44.259500 18841 leader_election.cc:304] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:44.260243 20461 raft_consensus.cc:3054] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:44.266891 18986 leader_election.cc:304] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:44.269009 20501 raft_consensus.cc:2798] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:44.269532 20501 raft_consensus.cc:695] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:44.270642 20501 consensus_queue.cc:237] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.273195 20512 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250111 02:57:44.294895 19131 leader_election.cc:304] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:44.295861 20509 raft_consensus.cc:2798] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:44.296255 20509 raft_consensus.cc:491] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:44.296597 20509 raft_consensus.cc:3054] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:44.296715 20296 raft_consensus.cc:513] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.298624 20296 leader_election.cc:290] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:44.299710 18903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c669a3ff8aa46fa8b04b86ed3bfde49" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:44.300882 18989 leader_election.cc:343] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:44.306381 20461 raft_consensus.cc:2743] T d83cfd85a52f438dabe49b6223d5090f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:44.314138 19330 tablet_bootstrap.cc:492] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:44.314628 19330 ts_tablet_manager.cc:1397] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.172s user 0.009s sys 0.001s
I20250111 02:57:44.320070 19322 tablet_bootstrap.cc:492] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:44.320535 19322 ts_tablet_manager.cc:1397] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.223s user 0.010s sys 0.002s
I20250111 02:57:44.325502 20509 raft_consensus.cc:513] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:44.342612 18897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076aafe3958948079e42501c6bd86153" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:44.344553 19133 leader_election.cc:343] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:44.354068 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c669a3ff8aa46fa8b04b86ed3bfde49" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:44.354918 19197 raft_consensus.cc:3054] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:44.362253 19330 raft_consensus.cc:357] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.375849 19322 raft_consensus.cc:357] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.376765 19322 raft_consensus.cc:383] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:44.377115 19322 raft_consensus.cc:738] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:44.377094 20509 leader_election.cc:290] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:44.377835 19322 consensus_queue.cc:260] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.380240 19322 ts_tablet_manager.cc:1428] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.059s user 0.006s sys 0.000s
I20250111 02:57:44.385226 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076aafe3958948079e42501c6bd86153" candidate_uuid: "d0cedab82b154d62b78809edc6f28bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:44.400081 19330 raft_consensus.cc:383] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:44.400282 19048 raft_consensus.cc:3054] T 076aafe3958948079e42501c6bd86153 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:44.403424 19197 raft_consensus.cc:2463] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:44.427690 19048 raft_consensus.cc:2463] T 076aafe3958948079e42501c6bd86153 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0cedab82b154d62b78809edc6f28bfa in term 1.
I20250111 02:57:44.478927 19322 tablet_bootstrap.cc:492] T aaab428f798a44f69c80e5792d56b920 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:44.483008 20461 raft_consensus.cc:491] T 64abb30da7fc452cad774bbe59b498f4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:44.483525 20461 raft_consensus.cc:513] T 64abb30da7fc452cad774bbe59b498f4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.485376 20461 leader_election.cc:290] T 64abb30da7fc452cad774bbe59b498f4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:44.486119 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64abb30da7fc452cad774bbe59b498f4" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:44.414847 18986 consensus_peers.cc:487] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cd364f9a6894d19826deb59a8d8a612. This is attempt 6: this message will repeat every 5th retry.
W20250111 02:57:44.495500 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 80793.3us
I20250111 02:57:44.496171 18986 leader_election.cc:304] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:44.435164 19330 raft_consensus.cc:738] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:44.486449 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64abb30da7fc452cad774bbe59b498f4" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:44.498904 19330 consensus_queue.cc:260] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.500572 18842 leader_election.cc:304] T 64abb30da7fc452cad774bbe59b498f4 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:44.501101 19131 leader_election.cc:304] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:44.501407 20461 raft_consensus.cc:2743] T 64abb30da7fc452cad774bbe59b498f4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:44.501855 20509 raft_consensus.cc:2798] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election won for term 1
W20250111 02:57:44.501892 19109 tablet.cc:2367] T 076aafe3958948079e42501c6bd86153 P 54628fbcec2f4359aa9e26a734d4d29f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:57:44.502295 20509 raft_consensus.cc:695] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [term 1 LEADER]: Becoming Leader. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Running, Role: LEADER
I20250111 02:57:44.503352 20509 consensus_queue.cc:237] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:44.511286 20296 raft_consensus.cc:2798] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:44.511835 20296 raft_consensus.cc:695] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:44.512627 20296 consensus_queue.cc:237] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:44.516897 19594 leader_election.cc:336] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.653s (SENT)
I20250111 02:57:44.517370 19594 leader_election.cc:304] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727; no voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:44.518707 20487 raft_consensus.cc:491] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:44.519151 20487 raft_consensus.cc:513] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:44.521039 20487 leader_election.cc:290] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:44.524420 20402 consensus_queue.cc:1035] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:44.531788 19330 ts_tablet_manager.cc:1428] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.217s user 0.006s sys 0.005s
I20250111 02:57:44.532801 19330 tablet_bootstrap.cc:492] T d3b7616fe56f4e5fb49bdde6961fa008 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:44.533005 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31ad2e1895d4a26b8e9f7cd7366ef81" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:44.533471 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31ad2e1895d4a26b8e9f7cd7366ef81" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:44.534065 19192 raft_consensus.cc:2463] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
W20250111 02:57:44.542573 18989 leader_election.cc:343] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:44.559747 19330 tablet_bootstrap.cc:654] T d3b7616fe56f4e5fb49bdde6961fa008 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:44.582105 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 44957.2us
I20250111 02:57:44.582767 18986 leader_election.cc:304] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:44.583652 20401 raft_consensus.cc:2798] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:44.583992 20401 raft_consensus.cc:491] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:44.584307 20401 raft_consensus.cc:3054] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:44.608341 20401 raft_consensus.cc:513] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:44.620086 19322 tablet_bootstrap.cc:654] T aaab428f798a44f69c80e5792d56b920 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:44.627941 20517 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250111 02:57:44.628742 19594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 111892us
I20250111 02:57:44.644493 20518 consensus_queue.cc:1035] T d83cfd85a52f438dabe49b6223d5090f P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:44.647575 20401 leader_election.cc:290] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:44.629235 19330 tablet_bootstrap.cc:492] T d3b7616fe56f4e5fb49bdde6961fa008 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:44.648492 19330 ts_tablet_manager.cc:1397] T d3b7616fe56f4e5fb49bdde6961fa008 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.116s user 0.011s sys 0.000s
I20250111 02:57:44.631847 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31ad2e1895d4a26b8e9f7cd7366ef81" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:44.667589 18989 leader_election.cc:343] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:44.669616 18899 raft_consensus.cc:3054] T 64abb30da7fc452cad774bbe59b498f4 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:44.698303 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31ad2e1895d4a26b8e9f7cd7366ef81" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:44.699033 19193 raft_consensus.cc:3054] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:44.650900 19330 raft_consensus.cc:357] T d3b7616fe56f4e5fb49bdde6961fa008 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.699797 19330 raft_consensus.cc:383] T d3b7616fe56f4e5fb49bdde6961fa008 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:44.700109 19330 raft_consensus.cc:738] T d3b7616fe56f4e5fb49bdde6961fa008 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:44.700866 19330 consensus_queue.cc:260] T d3b7616fe56f4e5fb49bdde6961fa008 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.703084 19330 ts_tablet_manager.cc:1428] T d3b7616fe56f4e5fb49bdde6961fa008 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.054s user 0.005s sys 0.000s
I20250111 02:57:44.704089 19330 tablet_bootstrap.cc:492] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:44.706549 19193 raft_consensus.cc:2463] T f31ad2e1895d4a26b8e9f7cd7366ef81 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:44.724004 18986 leader_election.cc:304] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:44.724817 20401 raft_consensus.cc:2798] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:44.725322 20401 raft_consensus.cc:695] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:44.726848 20401 consensus_queue.cc:237] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:44.733821 19322 tablet_bootstrap.cc:492] T aaab428f798a44f69c80e5792d56b920 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:44.734691 19322 ts_tablet_manager.cc:1397] T aaab428f798a44f69c80e5792d56b920 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.256s user 0.010s sys 0.001s
I20250111 02:57:44.737380 19322 raft_consensus.cc:357] T aaab428f798a44f69c80e5792d56b920 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.739249 19322 raft_consensus.cc:383] T aaab428f798a44f69c80e5792d56b920 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250111 02:57:44.760604 18989 consensus_peers.cc:487] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:44.790365 19322 raft_consensus.cc:738] T aaab428f798a44f69c80e5792d56b920 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:44.791257 19322 consensus_queue.cc:260] T aaab428f798a44f69c80e5792d56b920 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.794025 19322 ts_tablet_manager.cc:1428] T aaab428f798a44f69c80e5792d56b920 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.059s user 0.006s sys 0.000s
I20250111 02:57:44.809515 19330 tablet_bootstrap.cc:654] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:44.837308 19322 tablet_bootstrap.cc:492] T 6ff06134303d4cb0b4292fe1c0c2d443 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:44.867213 20401 consensus_queue.cc:1035] T 07f15096ea484b499fbdecec74eb6f66 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:44.912838 19322 tablet_bootstrap.cc:654] T 6ff06134303d4cb0b4292fe1c0c2d443 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:44.923799 20486 raft_consensus.cc:491] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:44.924300 20486 raft_consensus.cc:513] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.926383 20486 leader_election.cc:290] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:44.929126 18842 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: 32afc2f158254811836fcf00f7f409ed
W20250111 02:57:44.929718 18841 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Not found: Tablet not found: 32afc2f158254811836fcf00f7f409ed
I20250111 02:57:44.930112 18841 leader_election.cc:304] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:44.931120 20486 raft_consensus.cc:2743] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:44.956264 20402 raft_consensus.cc:491] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:44.956753 20402 raft_consensus.cc:513] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.958694 20402 leader_election.cc:290] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:44.960311 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaab428f798a44f69c80e5792d56b920" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:44.961197 18899 raft_consensus.cc:2463] T aaab428f798a44f69c80e5792d56b920 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:44.962553 18989 leader_election.cc:304] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:44.963405 20401 raft_consensus.cc:2798] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:44.963804 20401 raft_consensus.cc:491] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:44.964197 20401 raft_consensus.cc:3054] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:44.976917 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaab428f798a44f69c80e5792d56b920" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:44.977725 19193 raft_consensus.cc:2463] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:44.978914 19330 tablet_bootstrap.cc:492] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:44.979457 19330 ts_tablet_manager.cc:1397] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.276s user 0.012s sys 0.002s
I20250111 02:57:44.982031 19330 raft_consensus.cc:357] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:44.983134 19330 raft_consensus.cc:383] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:44.983462 19330 raft_consensus.cc:738] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:44.984268 19330 consensus_queue.cc:260] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.006556 19322 tablet_bootstrap.cc:492] T 6ff06134303d4cb0b4292fe1c0c2d443 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:45.007159 19322 ts_tablet_manager.cc:1397] T 6ff06134303d4cb0b4292fe1c0c2d443 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.170s user 0.012s sys 0.000s
I20250111 02:57:45.010125 19322 raft_consensus.cc:357] T 6ff06134303d4cb0b4292fe1c0c2d443 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.010905 19322 raft_consensus.cc:383] T 6ff06134303d4cb0b4292fe1c0c2d443 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:45.011234 19322 raft_consensus.cc:738] T 6ff06134303d4cb0b4292fe1c0c2d443 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:45.011936 19322 consensus_queue.cc:260] T 6ff06134303d4cb0b4292fe1c0c2d443 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.016785 19330 ts_tablet_manager.cc:1428] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.037s user 0.006s sys 0.000s
I20250111 02:57:45.044517 19322 ts_tablet_manager.cc:1428] T 6ff06134303d4cb0b4292fe1c0c2d443 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.037s user 0.005s sys 0.000s
I20250111 02:57:45.049036 20401 raft_consensus.cc:513] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.052078 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaab428f798a44f69c80e5792d56b920" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:45.054471 18899 raft_consensus.cc:3054] T aaab428f798a44f69c80e5792d56b920 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:45.062263 18899 raft_consensus.cc:2463] T aaab428f798a44f69c80e5792d56b920 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:45.063978 19330 tablet_bootstrap.cc:492] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
W20250111 02:57:45.064899 18989 consensus_peers.cc:487] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:45.066037 18989 leader_election.cc:304] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, 8221604733fe4fc0a1fc32de8d6af71c; no voters:
I20250111 02:57:45.066911 20518 raft_consensus.cc:2798] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:45.067389 20518 raft_consensus.cc:695] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:45.067900 19322 tablet_bootstrap.cc:492] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:45.068176 20518 consensus_queue.cc:237] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.070092 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaab428f798a44f69c80e5792d56b920" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:45.070705 19192 raft_consensus.cc:3054] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:45.073603 19330 tablet_bootstrap.cc:654] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:45.079470 19192 raft_consensus.cc:2463] T aaab428f798a44f69c80e5792d56b920 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:45.068588 20401 leader_election.cc:290] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:45.106138 19322 tablet_bootstrap.cc:654] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:45.134536 20536 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:45.135200 20536 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:45.138649 19330 tablet_bootstrap.cc:492] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:45.138644 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:45.139211 19330 ts_tablet_manager.cc:1397] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.075s user 0.011s sys 0.001s
I20250111 02:57:45.151684 19330 raft_consensus.cc:357] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.154049 19330 raft_consensus.cc:383] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:45.156255 19330 raft_consensus.cc:738] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:45.159638 19330 consensus_queue.cc:260] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.184439 20536 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:45.186673 19330 ts_tablet_manager.cc:1428] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.047s user 0.007s sys 0.000s
I20250111 02:57:45.191900 19322 tablet_bootstrap.cc:492] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:45.192339 19322 ts_tablet_manager.cc:1397] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.127s user 0.000s sys 0.012s
I20250111 02:57:45.194543 19322 raft_consensus.cc:357] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.198877 19322 raft_consensus.cc:383] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:45.199224 19322 raft_consensus.cc:738] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:45.199872 19322 consensus_queue.cc:260] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.202277 19322 ts_tablet_manager.cc:1428] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.010s user 0.004s sys 0.001s
I20250111 02:57:45.214684 19330 tablet_bootstrap.cc:492] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:45.220350 19330 tablet_bootstrap.cc:654] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:45.257642 19322 tablet_bootstrap.cc:492] T 3514d795a0f24adcaa2accc0cf05b142 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:45.268275 19330 tablet_bootstrap.cc:492] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:45.268795 19330 ts_tablet_manager.cc:1397] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.000s
I20250111 02:57:45.271764 19330 raft_consensus.cc:357] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:45.272497 19330 raft_consensus.cc:383] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:45.272789 19330 raft_consensus.cc:738] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:45.273506 19330 consensus_queue.cc:260] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:45.284934 19330 ts_tablet_manager.cc:1428] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20250111 02:57:45.323379 20296 consensus_queue.cc:1035] T 48dd0bccf6854d37a7ef95b5ab06308e P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:45.336146 19330 tablet_bootstrap.cc:492] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:45.346171 19322 tablet_bootstrap.cc:654] T 3514d795a0f24adcaa2accc0cf05b142 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:45.354038 20296 consensus_queue.cc:1035] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:45.370004 20486 raft_consensus.cc:491] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:45.370504 20486 raft_consensus.cc:513] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.372574 20486 leader_election.cc:290] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:45.375224 19187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c669a3ff8aa46fa8b04b86ed3bfde49" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:45.375473 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c669a3ff8aa46fa8b04b86ed3bfde49" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:45.376035 19187 raft_consensus.cc:2388] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:45.377156 18841 leader_election.cc:304] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:45.378445 20486 raft_consensus.cc:3054] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:45.428434 20541 log.cc:926] Time spent T a6e4b222bf974a8d80b25141a5686761 P d0cedab82b154d62b78809edc6f28bfa: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20250111 02:57:45.431708 20486 raft_consensus.cc:2743] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:45.440151 20402 consensus_queue.cc:1035] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250111 02:57:45.449579 18989 consensus_peers.cc:487] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:45.460451 20461 raft_consensus.cc:491] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:45.460964 20461 raft_consensus.cc:513] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.465628 20461 leader_election.cc:290] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:45.465685 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b7616fe56f4e5fb49bdde6961fa008" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:45.466282 19192 raft_consensus.cc:2463] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:45.467531 18841 leader_election.cc:304] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:45.468519 20486 raft_consensus.cc:2798] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:45.468941 20486 raft_consensus.cc:491] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:45.469099 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b7616fe56f4e5fb49bdde6961fa008" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:45.469444 20486 raft_consensus.cc:3054] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:45.469740 19048 raft_consensus.cc:2463] T d3b7616fe56f4e5fb49bdde6961fa008 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 0.
I20250111 02:57:45.485535 19322 tablet_bootstrap.cc:492] T 3514d795a0f24adcaa2accc0cf05b142 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:45.486021 19322 ts_tablet_manager.cc:1397] T 3514d795a0f24adcaa2accc0cf05b142 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.283s user 0.007s sys 0.002s
I20250111 02:57:45.478928 20486 raft_consensus.cc:513] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.488996 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b7616fe56f4e5fb49bdde6961fa008" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:45.489748 19193 raft_consensus.cc:3054] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:45.490278 20401 raft_consensus.cc:491] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:45.490690 20401 raft_consensus.cc:513] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.490629 19322 raft_consensus.cc:357] T 3514d795a0f24adcaa2accc0cf05b142 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.490787 20509 consensus_queue.cc:1035] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:45.491344 19322 raft_consensus.cc:383] T 3514d795a0f24adcaa2accc0cf05b142 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:45.491585 19322 raft_consensus.cc:738] T 3514d795a0f24adcaa2accc0cf05b142 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:45.493353 20401 leader_election.cc:290] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:45.495465 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3514d795a0f24adcaa2accc0cf05b142" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:45.496088 19192 raft_consensus.cc:2463] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:45.497179 18986 leader_election.cc:304] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:45.497938 20401 raft_consensus.cc:2798] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:45.498306 20401 raft_consensus.cc:491] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:45.498646 20401 raft_consensus.cc:3054] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:45.498919 20486 leader_election.cc:290] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Requested vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:45.500356 19193 raft_consensus.cc:2463] T d3b7616fe56f4e5fb49bdde6961fa008 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:45.503484 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b7616fe56f4e5fb49bdde6961fa008" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f"
I20250111 02:57:45.504099 19051 raft_consensus.cc:3054] T d3b7616fe56f4e5fb49bdde6961fa008 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:45.507838 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3514d795a0f24adcaa2accc0cf05b142" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:45.508039 18841 leader_election.cc:304] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:45.508836 20486 raft_consensus.cc:2798] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:45.509907 20461 raft_consensus.cc:491] T 6ff06134303d4cb0b4292fe1c0c2d443 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:45.510411 20461 raft_consensus.cc:513] T 6ff06134303d4cb0b4292fe1c0c2d443 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.513195 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff06134303d4cb0b4292fe1c0c2d443" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:45.515156 20461 leader_election.cc:290] T 6ff06134303d4cb0b4292fe1c0c2d443 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:45.515172 20402 raft_consensus.cc:491] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:45.509593 20486 raft_consensus.cc:695] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 LEADER]: Becoming Leader. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Running, Role: LEADER
I20250111 02:57:45.515666 20402 raft_consensus.cc:513] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.516438 20486 consensus_queue.cc:237] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.517827 20402 leader_election.cc:290] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:45.519766 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff06134303d4cb0b4292fe1c0c2d443" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:45.520865 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f4500d3beb4c9bb20f84c661c3758b" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:45.522058 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f4500d3beb4c9bb20f84c661c3758b" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:45.522151 18989 leader_election.cc:343] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:45.522770 19193 raft_consensus.cc:2463] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:45.524251 18842 leader_election.cc:304] T 6ff06134303d4cb0b4292fe1c0c2d443 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:45.524369 18986 leader_election.cc:304] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:45.525077 20461 raft_consensus.cc:2743] T 6ff06134303d4cb0b4292fe1c0c2d443 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:45.526082 20296 raft_consensus.cc:2798] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:45.526436 20296 raft_consensus.cc:491] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:45.526744 20296 raft_consensus.cc:3054] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:45.534616 20401 raft_consensus.cc:513] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.554067 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3514d795a0f24adcaa2accc0cf05b142" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:45.558490 20296 raft_consensus.cc:513] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.492197 19322 consensus_queue.cc:260] T 3514d795a0f24adcaa2accc0cf05b142 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:45.567363 18910 raft_consensus.cc:2463] T 3514d795a0f24adcaa2accc0cf05b142 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:45.568895 19322 ts_tablet_manager.cc:1428] T 3514d795a0f24adcaa2accc0cf05b142 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.083s user 0.007s sys 0.000s
I20250111 02:57:45.571270 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3514d795a0f24adcaa2accc0cf05b142" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:45.571919 19193 raft_consensus.cc:3054] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:45.580853 19193 raft_consensus.cc:2463] T 3514d795a0f24adcaa2accc0cf05b142 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:45.583039 18986 leader_election.cc:304] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:45.584580 20296 leader_election.cc:290] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:45.586450 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f4500d3beb4c9bb20f84c661c3758b" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:45.588016 20546 log.cc:926] Time spent T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 8221604733fe4fc0a1fc32de8d6af71c: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20250111 02:57:45.604420 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f4500d3beb4c9bb20f84c661c3758b" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:45.605108 19193 raft_consensus.cc:3054] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:45.569988 20401 leader_election.cc:290] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:45.565137 19051 raft_consensus.cc:2463] T d3b7616fe56f4e5fb49bdde6961fa008 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221604733fe4fc0a1fc32de8d6af71c in term 1.
I20250111 02:57:45.585283 20487 raft_consensus.cc:2798] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
W20250111 02:57:45.607555 19133 consensus_peers.cc:487] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:45.607991 20487 raft_consensus.cc:695] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:45.608830 20487 consensus_queue.cc:237] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:45.588094 18989 leader_election.cc:343] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:45.620860 19330 tablet_bootstrap.cc:654] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:45.630743 20551 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:45.631573 20551 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:45.570071 19322 tablet_bootstrap.cc:492] T a6e4b222bf974a8d80b25141a5686761 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:45.679603 20487 consensus_queue.cc:1035] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:45.700518 19193 raft_consensus.cc:2463] T 79f4500d3beb4c9bb20f84c661c3758b P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:45.713608 20296 consensus_queue.cc:1035] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:45.732355 19330 tablet_bootstrap.cc:492] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:45.732868 19330 ts_tablet_manager.cc:1397] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.397s user 0.017s sys 0.003s
I20250111 02:57:45.735473 19330 raft_consensus.cc:357] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:45.755851 19330 raft_consensus.cc:383] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:45.759975 19330 raft_consensus.cc:738] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:45.762369 19330 consensus_queue.cc:260] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:45.773455 20551 leader_election.cc:290] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8784b9603f8544b79373158aace063af (127.18.20.125:40359), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:45.773797 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:45.774426 18713 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e336d4ddf044bf5b0b6e4f5af634727 in term 8.
I20250111 02:57:45.776041 19595 leader_election.cc:304] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c; no voters:
I20250111 02:57:45.780077 19330 ts_tablet_manager.cc:1428] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.047s user 0.003s sys 0.002s
W20250111 02:57:45.794543 18989 consensus_peers.cc:487] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:45.795547 20551 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250111 02:57:45.795940 20551 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:45.796308 20551 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 8 FOLLOWER]: Advancing to term 9
I20250111 02:57:45.801744 19322 tablet_bootstrap.cc:654] T a6e4b222bf974a8d80b25141a5686761 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:45.837606 18986 leader_election.cc:304] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:45.839334 20401 raft_consensus.cc:2798] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:45.839766 20401 raft_consensus.cc:695] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:45.840444 20401 consensus_queue.cc:237] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:45.864050 18693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 80475.9us
I20250111 02:57:45.874416 20401 consensus_queue.cc:1035] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:45.952356 18897 raft_consensus.cc:3054] T 6ff06134303d4cb0b4292fe1c0c2d443 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:45.973623 20518 consensus_queue.cc:1035] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:45.997762 20551 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:46.013417 19322 tablet_bootstrap.cc:492] T a6e4b222bf974a8d80b25141a5686761 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:46.013911 19322 ts_tablet_manager.cc:1397] T a6e4b222bf974a8d80b25141a5686761 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.444s user 0.013s sys 0.000s
I20250111 02:57:46.019735 19322 raft_consensus.cc:357] T a6e4b222bf974a8d80b25141a5686761 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:46.020511 19322 raft_consensus.cc:383] T a6e4b222bf974a8d80b25141a5686761 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:46.020803 19322 raft_consensus.cc:738] T a6e4b222bf974a8d80b25141a5686761 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:46.021435 19322 consensus_queue.cc:260] T a6e4b222bf974a8d80b25141a5686761 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:46.062369 20552 raft_consensus.cc:491] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:46.062829 20552 raft_consensus.cc:513] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:46.064726 20552 leader_election.cc:290] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:46.068185 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c57d85390324bbc9ea30a709faec01c" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:46.068842 19192 raft_consensus.cc:2463] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:46.070261 18986 leader_election.cc:304] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:46.071529 20552 raft_consensus.cc:2798] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:46.071928 20552 raft_consensus.cc:491] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:46.072450 20552 raft_consensus.cc:3054] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:46.080592 20552 raft_consensus.cc:513] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:46.140448 20551 leader_election.cc:290] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 election: Requested vote from peers 8784b9603f8544b79373158aace063af (127.18.20.125:40359), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:46.140938 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c"
I20250111 02:57:46.066937 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c57d85390324bbc9ea30a709faec01c" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
I20250111 02:57:46.229590 19322 ts_tablet_manager.cc:1428] T a6e4b222bf974a8d80b25141a5686761 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.215s user 0.005s sys 0.000s
I20250111 02:57:46.218966 20296 raft_consensus.cc:491] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:46.245299 20296 raft_consensus.cc:513] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:46.247231 20296 leader_election.cc:290] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:46.227960 18989 leader_election.cc:343] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:46.249627 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56fb0bf949904208af003f08caae7594" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c" is_pre_election: true
W20250111 02:57:46.251354 18989 leader_election.cc:343] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:46.268179 19683 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 9 FOLLOWER]: Rejecting Update request from peer 8784b9603f8544b79373158aace063af for earlier term 8. Current term is 9. Ops: []
I20250111 02:57:46.270395 20523 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 }, Status: INVALID_TERM, Last received: 8.281, Next index: 282, Last known committed idx: 281, Time since last communication: 0.001s
I20250111 02:57:46.272650 20523 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 8 LEADER]: Stepping down as leader of term 8
I20250111 02:57:46.273140 20523 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 8784b9603f8544b79373158aace063af, State: Running, Role: LEADER
I20250111 02:57:46.274129 20523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 8.281, Last appended by leader: 281, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:46.275641 20523 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 8 FOLLOWER]: Advancing to term 9
I20250111 02:57:46.289510 19192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56fb0bf949904208af003f08caae7594" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:46.290226 19192 raft_consensus.cc:2463] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 0.
I20250111 02:57:46.305440 20552 leader_election.cc:290] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:46.315805 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c57d85390324bbc9ea30a709faec01c" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
W20250111 02:57:46.317257 18989 leader_election.cc:343] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:46.326526 19322 tablet_bootstrap.cc:492] T 076aafe3958948079e42501c6bd86153 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
W20250111 02:57:46.334625 18986 consensus_peers.cc:487] T 62c6b166a29943ceb15f093fac728c79 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62c6b166a29943ceb15f093fac728c79. This is attempt 6: this message will repeat every 5th retry.
I20250111 02:57:46.336158 18986 leader_election.cc:304] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
W20250111 02:57:46.392761 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 56682.4us
I20250111 02:57:46.393456 20552 raft_consensus.cc:2798] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:57:46.422714 20552 raft_consensus.cc:491] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:46.423252 20552 raft_consensus.cc:3054] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:46.437103 18614 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet a05dc653a5384a50b54e3568798633e8 (table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet a05dc653a5384a50b54e3568798633e8 consensus state has no leader
I20250111 02:57:46.442560 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c57d85390324bbc9ea30a709faec01c" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:46.443213 19193 raft_consensus.cc:3054] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:46.451951 19193 raft_consensus.cc:2463] T 3c57d85390324bbc9ea30a709faec01c P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
W20250111 02:57:46.462149 18693 leader_election.cc:336] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.596s (SENT)
I20250111 02:57:46.462606 18693 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:46.479221 20570 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250111 02:57:46.487778 18986 leader_election.cc:304] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters: 8221604733fe4fc0a1fc32de8d6af71c
I20250111 02:57:46.502528 20518 raft_consensus.cc:2798] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:57:46.503103 20518 raft_consensus.cc:695] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:46.509490 20518 consensus_queue.cc:237] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:46.552426 18908 raft_consensus.cc:3054] T 3514d795a0f24adcaa2accc0cf05b142 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:46.575325 18897 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.092s sys 0.124s
W20250111 02:57:46.575829 18897 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s user 0.092s sys 0.124s
I20250111 02:57:46.597755 20571 consensus_queue.cc:1035] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:46.616279 20552 raft_consensus.cc:513] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:46.630460 18897 raft_consensus.cc:3054] T a6e4b222bf974a8d80b25141a5686761 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:46.634649 19322 tablet_bootstrap.cc:654] T 076aafe3958948079e42501c6bd86153 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:46.645506 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 117778us
W20250111 02:57:46.694250 19594 leader_election.cc:336] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.521s (SENT)
I20250111 02:57:46.701920 20552 leader_election.cc:290] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Requested vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:46.720798 19187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56fb0bf949904208af003f08caae7594" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa"
I20250111 02:57:46.721490 19187 raft_consensus.cc:3054] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:46.732203 19187 raft_consensus.cc:2463] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54628fbcec2f4359aa9e26a734d4d29f in term 1.
I20250111 02:57:46.768184 19322 tablet_bootstrap.cc:492] T 076aafe3958948079e42501c6bd86153 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:46.768702 19322 ts_tablet_manager.cc:1397] T 076aafe3958948079e42501c6bd86153 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.442s user 0.014s sys 0.000s
I20250111 02:57:46.773013 19322 raft_consensus.cc:357] T 076aafe3958948079e42501c6bd86153 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:46.773819 19322 raft_consensus.cc:383] T 076aafe3958948079e42501c6bd86153 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:46.774103 19322 raft_consensus.cc:738] T 076aafe3958948079e42501c6bd86153 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:46.774698 19322 consensus_queue.cc:260] T 076aafe3958948079e42501c6bd86153 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:46.778218 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56fb0bf949904208af003f08caae7594" candidate_uuid: "54628fbcec2f4359aa9e26a734d4d29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8221604733fe4fc0a1fc32de8d6af71c"
I20250111 02:57:46.779304 19322 ts_tablet_manager.cc:1428] T 076aafe3958948079e42501c6bd86153 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.010s user 0.002s sys 0.004s
I20250111 02:57:46.780406 19322 tablet_bootstrap.cc:492] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
W20250111 02:57:46.786037 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 60951.2us
I20250111 02:57:46.810127 20523 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:46.810698 20523 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:46.814781 20523 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:46.814826 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 10 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:46.815512 18713 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 8.
I20250111 02:57:46.816850 18615 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c; no voters:
I20250111 02:57:46.819257 20523 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250111 02:57:46.819720 20523 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:46.820153 20523 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 9 FOLLOWER]: Advancing to term 10
I20250111 02:57:46.821928 20296 consensus_queue.cc:1035] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:46.824491 18986 leader_election.cc:304] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa; no voters:
I20250111 02:57:46.825191 20577 raft_consensus.cc:2798] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Leader election won for term 1
W20250111 02:57:46.828086 18989 consensus_peers.cc:487] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:46.828378 20577 raft_consensus.cc:695] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Becoming Leader. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Running, Role: LEADER
I20250111 02:57:46.829166 20577 consensus_queue.cc:237] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:46.832741 20487 consensus_queue.cc:1035] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:46.834920 20487 consensus_queue.cc:1035] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250111 02:57:46.836477 18989 leader_election.cc:343] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:57:46.843953 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 10 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:46.844558 19683 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 9.
I20250111 02:57:46.852378 19322 tablet_bootstrap.cc:654] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:46.860077 20523 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:46.878257 20571 consensus_queue.cc:1035] T d3b7616fe56f4e5fb49bdde6961fa008 P 8221604733fe4fc0a1fc32de8d6af71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:46.920503 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 10 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727"
I20250111 02:57:46.921146 19683 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 9 FOLLOWER]: Advancing to term 10
I20250111 02:57:46.935932 19322 tablet_bootstrap.cc:492] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:46.936513 19322 ts_tablet_manager.cc:1397] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.156s user 0.009s sys 0.008s
I20250111 02:57:46.939893 19322 raft_consensus.cc:357] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:46.940659 19322 raft_consensus.cc:383] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:46.940744 19683 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 10.
I20250111 02:57:46.940933 19322 raft_consensus.cc:738] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
W20250111 02:57:46.941844 20584 log.cc:926] Time spent T a6e4b222bf974a8d80b25141a5686761 P 8221604733fe4fc0a1fc32de8d6af71c: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20250111 02:57:46.962949 18616 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af; no voters:
I20250111 02:57:46.980077 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 10 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c"
I20250111 02:57:46.980719 18713 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 8 FOLLOWER]: Advancing to term 10
I20250111 02:57:46.987323 19322 consensus_queue.cc:260] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:46.989411 19322 ts_tablet_manager.cc:1428] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.053s user 0.005s sys 0.000s
I20250111 02:57:46.989822 20523 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 10 election: Requested vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:46.990607 19322 tablet_bootstrap.cc:492] T 48dd0bccf6854d37a7ef95b5ab06308e P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:46.995183 20523 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 10 FOLLOWER]: Leader election won for term 10
I20250111 02:57:47.005142 20523 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 10 LEADER]: Becoming Leader. State: Replica: 8784b9603f8544b79373158aace063af, State: Running, Role: LEADER
I20250111 02:57:47.005795 18713 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 10.
I20250111 02:57:47.017174 20523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 8.281, Last appended by leader: 281, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
W20250111 02:57:47.007350 18616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.126:45115, user_credentials={real_user=slave}} blocked reactor thread for 44618.4us
W20250111 02:57:47.053794 20592 log.cc:926] Time spent T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
I20250111 02:57:47.068003 20593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8784b9603f8544b79373158aace063af. Latest consensus state: current_term: 10 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:47.069428 20593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:47.153330 19322 tablet_bootstrap.cc:654] T 48dd0bccf6854d37a7ef95b5ab06308e P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:47.312350 19322 tablet_bootstrap.cc:492] T 48dd0bccf6854d37a7ef95b5ab06308e P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:47.312784 19322 ts_tablet_manager.cc:1397] T 48dd0bccf6854d37a7ef95b5ab06308e P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.322s user 0.010s sys 0.002s
I20250111 02:57:47.323647 20486 raft_consensus.cc:491] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:47.324203 20486 raft_consensus.cc:513] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:47.326213 20486 leader_election.cc:290] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:47.327353 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56fb0bf949904208af003f08caae7594" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:47.328092 19195 raft_consensus.cc:2388] T 56fb0bf949904208af003f08caae7594 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8221604733fe4fc0a1fc32de8d6af71c in current term 1: Already voted for candidate 54628fbcec2f4359aa9e26a734d4d29f in this term.
I20250111 02:57:47.327926 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56fb0bf949904208af003f08caae7594" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:47.329952 18841 leader_election.cc:304] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:47.330837 20486 raft_consensus.cc:3054] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:47.338398 20486 raft_consensus.cc:2743] T 56fb0bf949904208af003f08caae7594 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:47.340276 19322 raft_consensus.cc:357] T 48dd0bccf6854d37a7ef95b5ab06308e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:47.395829 19322 raft_consensus.cc:383] T 48dd0bccf6854d37a7ef95b5ab06308e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:47.396229 19322 raft_consensus.cc:738] T 48dd0bccf6854d37a7ef95b5ab06308e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:47.397014 19322 consensus_queue.cc:260] T 48dd0bccf6854d37a7ef95b5ab06308e P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:47.419843 19322 ts_tablet_manager.cc:1428] T 48dd0bccf6854d37a7ef95b5ab06308e P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.107s user 0.008s sys 0.000s
I20250111 02:57:47.429138 19322 tablet_bootstrap.cc:492] T 07f15096ea484b499fbdecec74eb6f66 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
W20250111 02:57:47.451107 19594 leader_election.cc:336] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.643s (SENT)
I20250111 02:57:47.451615 19594 leader_election.cc:304] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727; no voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:47.474277 18897 raft_consensus.cc:3054] T 076aafe3958948079e42501c6bd86153 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:47.508318 20598 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20250111 02:57:47.509326 19594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 58284.8us
I20250111 02:57:47.495040 19322 tablet_bootstrap.cc:654] T 07f15096ea484b499fbdecec74eb6f66 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:47.570863 18713 raft_consensus.cc:1270] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 10 FOLLOWER]: Refusing update from remote peer 8784b9603f8544b79373158aace063af: Log matching property violated. Preceding OpId in replica: term: 8 index: 281. Preceding OpId from leader: term: 10 index: 282. (index mismatch)
I20250111 02:57:47.572538 20593 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.001s
I20250111 02:57:47.587950 19683 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 10 FOLLOWER]: Refusing update from remote peer 8784b9603f8544b79373158aace063af: Log matching property violated. Preceding OpId in replica: term: 8 index: 281. Preceding OpId from leader: term: 10 index: 282. (index mismatch)
I20250111 02:57:47.590168 20593 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.001s
I20250111 02:57:47.611130 20602 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8784b9603f8544b79373158aace063af. Latest consensus state: current_term: 10 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:47.614085 20602 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:47.618328 20598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8784b9603f8544b79373158aace063af. Latest consensus state: current_term: 10 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:47.632369 20598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:47.642575 19322 tablet_bootstrap.cc:492] T 07f15096ea484b499fbdecec74eb6f66 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:47.658001 19322 ts_tablet_manager.cc:1397] T 07f15096ea484b499fbdecec74eb6f66 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.229s user 0.014s sys 0.000s
I20250111 02:57:47.661623 19322 raft_consensus.cc:357] T 07f15096ea484b499fbdecec74eb6f66 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:47.672533 18615 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 3fa8530e894c498cbbfa73a3ba1c751e (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3fa8530e894c498cbbfa73a3ba1c751e consensus state has no leader
I20250111 02:57:47.703094 20296 consensus_queue.cc:1035] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:47.733429 20602 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:47.734349 20602 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:47.767843 20598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:47.768937 20598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:47.796593 20593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:47.797566 20593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:47.798431 20593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:47.799551 20593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:47.829313 19322 raft_consensus.cc:383] T 07f15096ea484b499fbdecec74eb6f66 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:47.829716 19322 raft_consensus.cc:738] T 07f15096ea484b499fbdecec74eb6f66 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:47.854764 19322 consensus_queue.cc:260] T 07f15096ea484b499fbdecec74eb6f66 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:47.897305 18989 consensus_peers.cc:487] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:47.911717 20518 consensus_queue.cc:1035] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:57:47.942782 19322 ts_tablet_manager.cc:1428] T 07f15096ea484b499fbdecec74eb6f66 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.284s user 0.008s sys 0.000s
I20250111 02:57:47.943903 19322 tablet_bootstrap.cc:492] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:47.960886 18897 raft_consensus.cc:3054] T 07f15096ea484b499fbdecec74eb6f66 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:48.110864 19322 tablet_bootstrap.cc:654] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:48.112396 20579 consensus_queue.cc:1035] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:57:48.174707 20598 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:48.175558 20598 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:48.178048 20598 leader_election.cc:290] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 8784b9603f8544b79373158aace063af (127.18.20.125:40359), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:48.180953 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 11 candidate_status { last_received { term: 10 index: 282 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:48.185140 20602 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:48.185664 20602 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:48.187662 20602 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:48.189571 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 11 candidate_status { last_received { term: 10 index: 282 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:48.190251 19683 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c34b72b7c70343d3b6f3fae79ce4be6c in term 10.
I20250111 02:57:48.192710 18695 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c; no voters:
I20250111 02:57:48.194440 20602 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250111 02:57:48.194793 20602 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 10 FOLLOWER]: Starting leader election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:48.195127 20602 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 10 FOLLOWER]: Advancing to term 11
I20250111 02:57:48.205273 20602 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:48.208266 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 11 candidate_status { last_received { term: 10 index: 282 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727"
I20250111 02:57:48.208824 19683 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 10 FOLLOWER]: Advancing to term 11
I20250111 02:57:48.224659 19683 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c34b72b7c70343d3b6f3fae79ce4be6c in term 11.
I20250111 02:57:48.225126 20602 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 11 election: Requested vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:48.226009 18695 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, c34b72b7c70343d3b6f3fae79ce4be6c; no voters:
I20250111 02:57:48.236541 20602 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 11 FOLLOWER]: Leader election won for term 11
I20250111 02:57:48.272850 19683 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 11 FOLLOWER]: Rejecting Update request from peer 8784b9603f8544b79373158aace063af for earlier term 10. Current term is 11. Ops: []
I20250111 02:57:48.274420 20607 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 }, Status: INVALID_TERM, Last received: 10.282, Next index: 283, Last known committed idx: 282, Time since last communication: 0.001s
I20250111 02:57:48.275535 20593 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 10 LEADER]: Stepping down as leader of term 10
I20250111 02:57:48.275852 20593 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 8784b9603f8544b79373158aace063af, State: Running, Role: LEADER
I20250111 02:57:48.276600 20593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 10.282, Last appended by leader: 282, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:48.277880 20593 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 10 FOLLOWER]: Advancing to term 11
I20250111 02:57:48.295738 18897 raft_consensus.cc:3054] T 48dd0bccf6854d37a7ef95b5ab06308e P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:48.343690 20602 raft_consensus.cc:695] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 11 LEADER]: Becoming Leader. State: Replica: c34b72b7c70343d3b6f3fae79ce4be6c, State: Running, Role: LEADER
I20250111 02:57:48.344853 20602 consensus_queue.cc:237] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 10.282, Last appended by leader: 282, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:48.360390 20621 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c34b72b7c70343d3b6f3fae79ce4be6c. Latest consensus state: current_term: 11 leader_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:48.361631 20621 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:48.420962 19322 tablet_bootstrap.cc:492] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:48.421411 19322 ts_tablet_manager.cc:1397] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.478s user 0.010s sys 0.000s
I20250111 02:57:48.484679 19322 raft_consensus.cc:357] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:48.485402 19322 raft_consensus.cc:383] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:48.485688 19322 raft_consensus.cc:738] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:48.486248 19322 consensus_queue.cc:260] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:48.487924 19322 ts_tablet_manager.cc:1428] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.066s user 0.002s sys 0.005s
I20250111 02:57:48.488868 19322 tablet_bootstrap.cc:492] T 3c57d85390324bbc9ea30a709faec01c P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:48.494388 19322 tablet_bootstrap.cc:654] T 3c57d85390324bbc9ea30a709faec01c P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:48.515093 19322 tablet_bootstrap.cc:492] T 3c57d85390324bbc9ea30a709faec01c P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:48.515543 19322 ts_tablet_manager.cc:1397] T 3c57d85390324bbc9ea30a709faec01c P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.003s
I20250111 02:57:48.518386 19322 raft_consensus.cc:357] T 3c57d85390324bbc9ea30a709faec01c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:48.531385 19322 raft_consensus.cc:383] T 3c57d85390324bbc9ea30a709faec01c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:48.531726 19322 raft_consensus.cc:738] T 3c57d85390324bbc9ea30a709faec01c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:48.532384 19322 consensus_queue.cc:260] T 3c57d85390324bbc9ea30a709faec01c P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:48.534786 19322 ts_tablet_manager.cc:1428] T 3c57d85390324bbc9ea30a709faec01c P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250111 02:57:48.535761 19322 tablet_bootstrap.cc:492] T 79f4500d3beb4c9bb20f84c661c3758b P 8221604733fe4fc0a1fc32de8d6af71c: Bootstrap starting.
I20250111 02:57:48.559327 19322 tablet_bootstrap.cc:654] T 79f4500d3beb4c9bb20f84c661c3758b P 8221604733fe4fc0a1fc32de8d6af71c: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:48.599795 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 60210.6us
I20250111 02:57:48.601826 19322 tablet_bootstrap.cc:492] T 79f4500d3beb4c9bb20f84c661c3758b P 8221604733fe4fc0a1fc32de8d6af71c: No bootstrap required, opened a new log
I20250111 02:57:48.602254 19322 ts_tablet_manager.cc:1397] T 79f4500d3beb4c9bb20f84c661c3758b P 8221604733fe4fc0a1fc32de8d6af71c: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.004s
I20250111 02:57:48.602769 20623 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:57:48.605167 19322 raft_consensus.cc:357] T 79f4500d3beb4c9bb20f84c661c3758b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:48.605885 19322 raft_consensus.cc:383] T 79f4500d3beb4c9bb20f84c661c3758b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:48.606134 19322 raft_consensus.cc:738] T 79f4500d3beb4c9bb20f84c661c3758b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221604733fe4fc0a1fc32de8d6af71c, State: Initialized, Role: FOLLOWER
I20250111 02:57:48.615525 19322 consensus_queue.cc:260] T 79f4500d3beb4c9bb20f84c661c3758b P 8221604733fe4fc0a1fc32de8d6af71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:48.620735 19322 ts_tablet_manager.cc:1428] T 79f4500d3beb4c9bb20f84c661c3758b P 8221604733fe4fc0a1fc32de8d6af71c: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20250111 02:57:48.636749 20600 raft_consensus.cc:491] T 48dd0bccf6854d37a7ef95b5ab06308e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 54628fbcec2f4359aa9e26a734d4d29f)
I20250111 02:57:48.637516 20600 raft_consensus.cc:513] T 48dd0bccf6854d37a7ef95b5ab06308e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:48.646705 19187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dd0bccf6854d37a7ef95b5ab06308e" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:48.649000 20600 leader_election.cc:290] T 48dd0bccf6854d37a7ef95b5ab06308e P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
W20250111 02:57:48.667259 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 46701.5us
W20250111 02:57:48.668483 19132 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.20.126:45115, user_credentials={real_user=slave}} blocked reactor thread for 96341us
I20250111 02:57:48.675366 20593 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:48.675981 20593 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:48.678841 20593 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:48.680869 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 12 candidate_status { last_received { term: 10 index: 282 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:48.681525 19683 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 11.
I20250111 02:57:48.683864 18616 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af; no voters:
I20250111 02:57:48.684752 20593 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250111 02:57:48.685169 20593 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:57:48.685607 20593 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 11 FOLLOWER]: Advancing to term 12
I20250111 02:57:48.696674 20600 raft_consensus.cc:491] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:48.697573 20600 raft_consensus.cc:513] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:48.704275 19187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31ad2e1895d4a26b8e9f7cd7366ef81" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
W20250111 02:57:48.725982 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 57970.9us
I20250111 02:57:48.730293 20600 leader_election.cc:290] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591)
I20250111 02:57:48.755904 20600 raft_consensus.cc:491] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 54628fbcec2f4359aa9e26a734d4d29f)
I20250111 02:57:48.756929 20600 raft_consensus.cc:513] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:48.763199 18841 leader_election.cc:343] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Not found: Tablet not found: 4b302fcba27244c193586bc25fc9305f
I20250111 02:57:48.823261 20593 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:48.826697 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 12 candidate_status { last_received { term: 10 index: 282 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727"
I20250111 02:57:48.827309 19683 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 11 FOLLOWER]: Advancing to term 12
I20250111 02:57:48.835301 19683 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 12.
I20250111 02:57:48.837852 18616 leader_election.cc:304] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af; no voters:
W20250111 02:57:48.838053 18693 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.613s (SENT) [suppressed 1 similar messages]
W20250111 02:57:48.863462 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 65325.1us
I20250111 02:57:48.881124 19683 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 12 FOLLOWER]: Rejecting Update request from peer c34b72b7c70343d3b6f3fae79ce4be6c for earlier term 11. Current term is 12. Ops: []
I20250111 02:57:48.882911 20628 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 12 FOLLOWER]: Leader election won for term 12
I20250111 02:57:48.883011 20621 consensus_queue.cc:1046] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 }, Status: INVALID_TERM, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.001s
I20250111 02:57:48.887789 20600 leader_election.cc:290] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:48.914268 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dd0bccf6854d37a7ef95b5ab06308e" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:48.915187 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31ad2e1895d4a26b8e9f7cd7366ef81" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
W20250111 02:57:48.915525 18616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.126:45115, user_credentials={real_user=slave}} blocked reactor thread for 77742.5us
I20250111 02:57:48.932541 20632 raft_consensus.cc:3049] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 11 LEADER]: Stepping down as leader of term 11
I20250111 02:57:48.939549 20632 raft_consensus.cc:738] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 11 LEADER]: Becoming Follower/Learner. State: Replica: c34b72b7c70343d3b6f3fae79ce4be6c, State: Running, Role: LEADER
I20250111 02:57:48.940941 20632 consensus_queue.cc:260] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 11.283, Last appended by leader: 283, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:48.944037 20632 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 11 FOLLOWER]: Advancing to term 12
W20250111 02:57:48.945582 18617 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b54311f4269d4575b8998d7f545000e7 (table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet b54311f4269d4575b8998d7f545000e7 consensus state has no leader
W20250111 02:57:48.954005 18693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 115972us
I20250111 02:57:48.913069 20593 leader_election.cc:290] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 12 election: Requested vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:48.968616 20628 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 12 LEADER]: Becoming Leader. State: Replica: 8784b9603f8544b79373158aace063af, State: Running, Role: LEADER
I20250111 02:57:48.969735 20628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 10.282, Last appended by leader: 282, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:49.033360 20600 raft_consensus.cc:491] T 79f4500d3beb4c9bb20f84c661c3758b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:49.048354 18897 raft_consensus.cc:3054] T 3c57d85390324bbc9ea30a709faec01c P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:49.034049 20600 raft_consensus.cc:513] T 79f4500d3beb4c9bb20f84c661c3758b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:49.078590 20631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8784b9603f8544b79373158aace063af. Latest consensus state: current_term: 12 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:49.079622 20631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:49.081128 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f4500d3beb4c9bb20f84c661c3758b" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cedab82b154d62b78809edc6f28bfa" is_pre_election: true
I20250111 02:57:49.090165 18908 raft_consensus.cc:3054] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:57:49.092053 20600 leader_election.cc:290] T 79f4500d3beb4c9bb20f84c661c3758b P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:49.122225 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 100061us
W20250111 02:57:49.135496 19132 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.20.126:45115, user_credentials={real_user=slave}} blocked reactor thread for 136401us
W20250111 02:57:49.213644 18842 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 121869us
I20250111 02:57:49.217272 18842 leader_election.cc:304] T 48dd0bccf6854d37a7ef95b5ab06308e P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:49.219796 18842 leader_election.cc:304] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:49.220556 20486 raft_consensus.cc:2743] T 48dd0bccf6854d37a7ef95b5ab06308e P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250111 02:57:49.222311 18842 leader_election.cc:343] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: 4b302fcba27244c193586bc25fc9305f
I20250111 02:57:49.222796 18842 leader_election.cc:304] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:49.223563 20600 raft_consensus.cc:2743] T 4b302fcba27244c193586bc25fc9305f P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250111 02:57:49.410284 18615 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.20.124:34593 timed out after 0.729s (SENT) [suppressed 3 similar messages]
W20250111 02:57:49.411413 19594 leader_election.cc:336] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.598s (SENT)
I20250111 02:57:49.412034 19594 leader_election.cc:304] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727; no voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:49.465090 20642 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250111 02:57:49.478989 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 118801us
W20250111 02:57:49.470822 19594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 59481.4us
W20250111 02:57:49.481510 18615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.124:34593, user_credentials={real_user=slave}} blocked reactor thread for 71254.5us
I20250111 02:57:49.491467 20486 raft_consensus.cc:2743] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:49.580124 19683 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 12 FOLLOWER]: Refusing update from remote peer 8784b9603f8544b79373158aace063af: Log matching property violated. Preceding OpId in replica: term: 10 index: 282. Preceding OpId from leader: term: 12 index: 283. (index mismatch)
I20250111 02:57:49.584197 20628 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
W20250111 02:57:49.589071 18693 leader_election.cc:336] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.613s (SENT)
I20250111 02:57:49.637159 18908 raft_consensus.cc:3054] T 79f4500d3beb4c9bb20f84c661c3758b P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:57:49.746872 19131 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.66:44623, user_credentials={real_user=slave}} blocked reactor thread for 137501us
W20250111 02:57:49.762260 18693 leader_election.cc:336] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.704s (SENT)
I20250111 02:57:49.764272 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f4500d3beb4c9bb20f84c661c3758b" candidate_uuid: "8221604733fe4fc0a1fc32de8d6af71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54628fbcec2f4359aa9e26a734d4d29f" is_pre_election: true
I20250111 02:57:49.770650 20645 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:49.771435 20645 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:49.781926 19682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 13 candidate_status { last_received { term: 11 index: 283 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:49.794589 20645 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:49.827279 18842 leader_election.cc:304] T 79f4500d3beb4c9bb20f84c661c3758b P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:49.845515 20600 raft_consensus.cc:2743] T 79f4500d3beb4c9bb20f84c661c3758b P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:50.016042 20642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8784b9603f8544b79373158aace063af. Latest consensus state: current_term: 12 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:50.016961 20642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: FOLLOWER
W20250111 02:57:50.067452 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 84298us
I20250111 02:57:50.092520 20642 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:50.093061 20642 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:50.095806 20642 leader_election.cc:290] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 8784b9603f8544b79373158aace063af (127.18.20.125:40359), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
W20250111 02:57:50.161378 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 69864.8us
W20250111 02:57:50.171456 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 86948.9us
W20250111 02:57:50.173791 18614 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 92aa64b2de5746d1b12cc93a10c9d56e (table default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92aa64b2de5746d1b12cc93a10c9d56e consensus state has no leader
I20250111 02:57:50.204196 20651 raft_consensus.cc:491] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:50.204720 20651 raft_consensus.cc:513] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
I20250111 02:57:50.206702 20651 leader_election.cc:290] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:50.208271 18841 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Not found: Tablet not found: 32afc2f158254811836fcf00f7f409ed
W20250111 02:57:50.212946 18615 leader_election.cc:336] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593): Timed out: RequestConsensusVote RPC to 127.18.20.124:34593 timed out after 0.729s (SENT)
I20250111 02:57:50.216583 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_26836b21f0f84973b442a4966f97d1ee [id=0037843fe90a422399b837b548ff45b9]
I20250111 02:57:50.218722 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609]
I20250111 02:57:50.221042 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_7653ac6289d7467590ac03b43df1b8bf [id=043fd6ad4f5c4a2685af5bbecda90328]
W20250111 02:57:50.222167 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 53469.1us
I20250111 02:57:50.254817 20642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 12 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:50.255661 20642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:50.285254 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_4b9e85e01e1843ce831205177cbb2e0f [id=0a118da8c5d0408980cbd0b33e0c2ed0]
I20250111 02:57:50.286059 20653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 12 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:50.286761 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_28d700d6f8444cfeab3dc52dd2bec17f [id=0e5eefce9dfa453eb277f75189def89e]
I20250111 02:57:50.287027 20653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: LEADER
W20250111 02:57:50.314298 18615 leader_election.cc:336] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593): Timed out: RequestConsensusVote RPC to 127.18.20.124:34593 timed out after 0.695s (SENT)
I20250111 02:57:50.381630 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_bccba04b48cc408d845624ee7ed98e5f [id=1496848ec9594cad889312978436d9c4]
I20250111 02:57:50.383291 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_c398ec562f594aca8c1e62cde38c04f9 [id=14bc02cbe30c486782d8fa4396855a9a]
W20250111 02:57:50.481498 18842 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: 32afc2f158254811836fcf00f7f409ed
I20250111 02:57:50.482092 18842 leader_election.cc:304] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:50.482903 20651 raft_consensus.cc:2743] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:50.487293 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_b6421db891c148c688d25610559fe16f [id=190cfd157c8d4bd19a8f40cfd80925c1]
I20250111 02:57:50.488765 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_dc22147bd8ba4ea18bc4f41b25b277c0 [id=1d14d647d8c24e31a84ce6ac7e2c217e]
I20250111 02:57:50.490960 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_37a329a4875c463ea093ad5ab23e9cf6 [id=23a0f0c1aaa149c68390a0d0327ff2c5]
W20250111 02:57:50.703625 19594 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.607s (SENT) [suppressed 3 similar messages]
I20250111 02:57:50.602013 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_e42719b273bf4b8f96fa48d1a3d6c8d8 [id=27f7d429d88b45b8b76d416b9dc582db]
W20250111 02:57:50.765766 19594 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 62141.7us
I20250111 02:57:50.803241 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_c9582d98645645f3aa80a65f7652cba8 [id=2d5d8f8a7ade443f91a7da34add25a91]
I20250111 02:57:50.805837 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_a02982c32dcb409385f8f4e99b95423f [id=354a3a4b0b774fb9bc83623b2cb87f8a]
I20250111 02:57:50.813442 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]
I20250111 02:57:50.843227 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_baa2b0022e914aa5b485db90037be4af [id=3f31cb2959f340e3ae2656b63eaa52e6]
I20250111 02:57:50.872167 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_4c9aebb06d554107a5448d968fdb0436 [id=3fab00618e6548de85b5d20adee4e9ac]
I20250111 02:57:50.878116 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_1ab2b1f61ced4c97ad5e44f18b6e8699 [id=40eb021e024747f0a173bf87538b7281]
I20250111 02:57:50.898552 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_bdacd508498241408a549fdbc6704512 [id=46a029216fdb4496bf9df1d776ccdf7e]
I20250111 02:57:50.911330 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_a68194fdb909449f878695d09b81c753 [id=49aa9df32f8e4001b66d751832a64534]
I20250111 02:57:50.920691 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_bf50e4c0392544f8a24b861d3edaf46e [id=4b08fa373b0b460094c09b4d3353951c]
I20250111 02:57:50.926951 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_4f96c1244592444a994b5c5631eaaadc [id=4c817a432cf649799a2589f12a320ebd]
I20250111 02:57:50.929384 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_39f0114c91ef4d1c86ef0cf522e31e11 [id=5189b95a9b0e4b47ac0d7768e08e902b]
I20250111 02:57:50.949690 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_dacea9000ff840659052c41c94a8bfaf [id=51bd1567dc9c43b2877791b72c2c2046]
I20250111 02:57:50.952471 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_535081a0b0ff46998f523371377b75f9 [id=531ca04d38e9427eb955326135436b79]
I20250111 02:57:51.016465 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_97e52a18fbd44f189d46ba9b26cd9033 [id=566e89e7552f4ce08fba16fe299b9ccf]
I20250111 02:57:51.040365 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_0dcad07d56d843e59f351d933e625a21 [id=57e820dd99764a578d0c50f5f250aad4]
I20250111 02:57:51.104734 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]
W20250111 02:57:51.115712 18693 leader_election.cc:336] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.639s (SENT)
I20250111 02:57:51.121938 18693 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:51.125626 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_85a0376bcaf24d0081e780e8c19764b8 [id=5f97e9e0988848319a9c8aaca6aad982]
I20250111 02:57:51.131541 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_c2fa721b96ba4e9ca293ff4b5c455447 [id=64d78652fc6f44778133bd3157964db5]
I20250111 02:57:51.136793 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_f36240205653470e9a67400b787ce860 [id=69f6e7b3ec40423a9d0bef5487ffe84a]
I20250111 02:57:51.164397 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_52d947397a774f96a4fdb5087da4e19a [id=7196ac9c0c4d4dada6c94803e1220caa]
I20250111 02:57:51.168639 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_4e3a79a5d2184687a8c51fd2702c5fd0 [id=73c59c0d9864497fbc9f19e265d8283f]
W20250111 02:57:51.178484 18693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.20.125:40359, user_credentials={real_user=slave}} blocked reactor thread for 62818us
I20250111 02:57:51.191501 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_b9769d513090402bb596c949defad0df [id=76e72ea89f6e408fbaeda0efc7e79770]
I20250111 02:57:51.198772 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_6a0ed66485f640e9a2c11b62b5cbd239 [id=7b936eef4300409e908cc92b1453b500]
I20250111 02:57:51.202456 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_8eae98e1e49c4f6c9e52644256812eaa [id=7dfae414add74b6491f2f7595a551342]
I20250111 02:57:51.192865 20670 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250111 02:57:51.221532 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]
I20250111 02:57:51.238404 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_53d44be8eaeb4e59a8e3454aac56daca [id=8de61815471842e593a7d29efffc443c]
I20250111 02:57:51.240312 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0]
I20250111 02:57:51.266136 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_5e7c139097914dd7b7c8e55058d5290d [id=9143d1d3b8754ac78d615a1cc4601d86]
I20250111 02:57:51.272676 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605]
I20250111 02:57:51.280443 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_c0ddb0fbf2f347c0b82acf9060722ed2 [id=964e4311f5ed4578a6f7285a9d2bbafa]
I20250111 02:57:51.281956 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_02b5991089a34b0b8ee6039ccc092c93 [id=977f59b134154bc396bf42bc1e44e717]
I20250111 02:57:51.283376 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_98958a28faeb43e8a3b34fcf20d936aa [id=98e24956666a413cbf4564438a594ef0]
I20250111 02:57:51.284754 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_f9d19027fe814825a6c3906baef77ff3 [id=9a4fbc1c2c2c44c182f9935156885c5a]
I20250111 02:57:51.286135 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_857f04d0de7e47a4828db9a8997f1009 [id=9cf908e0c7d44b60b88a32cf2d6368a0]
W20250111 02:57:51.375439 19595 leader_election.cc:336] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593): Timed out: RequestConsensusVote RPC to 127.18.20.124:34593 timed out after 0.607s (SENT)
W20250111 02:57:51.376307 19594 leader_election.cc:336] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 8784b9603f8544b79373158aace063af (127.18.20.125:40359): Timed out: RequestConsensusVote RPC to 127.18.20.125:40359 timed out after 0.607s (SENT)
I20250111 02:57:51.376758 19594 leader_election.cc:304] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727; no voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:51.408824 20671 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250111 02:57:51.450624 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7]
I20250111 02:57:51.452103 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_1431a031eea3496b9764a373a012a367 [id=a0cdf8fe647d4c84a9ce97df9ce9a0ce]
I20250111 02:57:51.454247 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_fc5930652905494584fde76fff8629d8 [id=a25d2996d4d64d91980bb5a067f5b118]
I20250111 02:57:51.529209 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_bdf982e1d6904892ac08ff16ed20a636 [id=a2b6467b58754b85ab0481a9e8aa0ace]
I20250111 02:57:51.531006 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19]
I20250111 02:57:51.533567 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_ee18021e6a544ed6b86b624aec8c39be [id=a9538f81833c4c0d8d02b6271a67f711]
I20250111 02:57:51.535300 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_2c181f66f28741b7b388a7b563b7d2f5 [id=ac0e88a97cf740fd8df3e6cbd81888b5]
I20250111 02:57:51.538169 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3]
I20250111 02:57:51.541704 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_6ad0b827920c42c68113310190019df1 [id=b25314ef99fc4e0ba80c6391223d2a3d]
I20250111 02:57:51.543354 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_b19a186d285d4b5b845727344f7d155b [id=b28b429fe5844cd6ac6f4de349c9449b]
I20250111 02:57:51.544965 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0]
I20250111 02:57:51.546818 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_dfe42fac0a764052a8d2f6add8882b64 [id=b42908f2a2c14d0a84027cb7a1d775b7]
I20250111 02:57:51.550422 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]
I20250111 02:57:51.552731 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_24e5a7f819cc45dbb7efbd7286ad7005 [id=c04c27fb52054de185e1f3075f89a59a]
I20250111 02:57:51.555294 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_23a3f1c40c15458e93c2ef79a3c9e9e2 [id=c12add3740174f66996466994ebe9971]
I20250111 02:57:51.556852 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_bf5cf724093d4691a60379b87cfab2f0 [id=c151a212f267426a9cb6180c938a5815]
I20250111 02:57:51.559450 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_aea3cbca3247424c9a12e310135fe0b9 [id=c151ba8ccc454372b4ad7621b6bc7aeb]
I20250111 02:57:51.561071 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927]
I20250111 02:57:51.571195 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_26836b21f0f84973b442a4966f97d1ee [id=0037843fe90a422399b837b548ff45b9]
I20250111 02:57:51.576906 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609]
I20250111 02:57:51.579337 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_7653ac6289d7467590ac03b43df1b8bf [id=043fd6ad4f5c4a2685af5bbecda90328]
I20250111 02:57:51.581918 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_4b9e85e01e1843ce831205177cbb2e0f [id=0a118da8c5d0408980cbd0b33e0c2ed0]
I20250111 02:57:51.583019 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_cce40438ec8e4527b01088af6b7b0981 [id=cb42d3930a9a44bfb5dc40feb979d505]
I20250111 02:57:51.584789 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_28d700d6f8444cfeab3dc52dd2bec17f [id=0e5eefce9dfa453eb277f75189def89e]
I20250111 02:57:51.587335 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5]
I20250111 02:57:51.591302 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_d7e00ace61e545129a2e99093dba16db [id=d0e7184505c24fc5b9960e8b6d1c955d]
I20250111 02:57:51.593189 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_bccba04b48cc408d845624ee7ed98e5f [id=1496848ec9594cad889312978436d9c4]
I20250111 02:57:51.594856 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3]
I20250111 02:57:51.596738 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_c398ec562f594aca8c1e62cde38c04f9 [id=14bc02cbe30c486782d8fa4396855a9a]
I20250111 02:57:51.599136 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_6823aef149894f49822fe303c036446d [id=d5ffe760744748a98f80f25b8092f9fa]
I20250111 02:57:51.601893 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_b6421db891c148c688d25610559fe16f [id=190cfd157c8d4bd19a8f40cfd80925c1]
I20250111 02:57:51.603461 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_5e21879e67544c71af627f795fd6d940 [id=d6c0153a97cd42d49f534f48fc5b8287]
I20250111 02:57:51.605840 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_813d4e8bb0f34a46bf96ed3bee6ca86f [id=d790288a097a4416b68b93db6ac847fd]
I20250111 02:57:51.607734 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_dc22147bd8ba4ea18bc4f41b25b277c0 [id=1d14d647d8c24e31a84ce6ac7e2c217e]
I20250111 02:57:51.612068 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_23d632d4476c46b78d27619201589641 [id=db5ffbbe4fe7424f943fbaca17103d84]
I20250111 02:57:51.612983 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_37a329a4875c463ea093ad5ab23e9cf6 [id=23a0f0c1aaa149c68390a0d0327ff2c5]
I20250111 02:57:51.614557 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_ea068446b5434b3ca5f0afa8386c0655 [id=dc09e842add34ef099edfd1296255c4b]
I20250111 02:57:51.617316 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_e42719b273bf4b8f96fa48d1a3d6c8d8 [id=27f7d429d88b45b8b76d416b9dc582db]
I20250111 02:57:51.618991 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_a7b2b5af5dfb439a84159b75de0d03b1 [id=dc7854580f744d4083f6aa3f0f0f7bf2]
I20250111 02:57:51.621357 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_f9181ca352384593b93119fb3e0cd812 [id=e0ac9d601c464a3daaaa63c4cb90e471]
I20250111 02:57:51.644903 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_c9582d98645645f3aa80a65f7652cba8 [id=2d5d8f8a7ade443f91a7da34add25a91]
I20250111 02:57:51.658841 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_c4faffc9659848ea9bff581e8d88d40e [id=e709d8f708144de8a18d241f0e0306a0]
I20250111 02:57:51.660200 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_a02982c32dcb409385f8f4e99b95423f [id=354a3a4b0b774fb9bc83623b2cb87f8a]
I20250111 02:57:51.672585 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]
I20250111 02:57:51.674357 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_799d0b8328de40a7bacb58ea433fec72 [id=e9efdc6c94574ddca092dd6b9f3b8419]
I20250111 02:57:51.677348 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_baa2b0022e914aa5b485db90037be4af [id=3f31cb2959f340e3ae2656b63eaa52e6]
I20250111 02:57:51.681756 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_902d917b19634ad5b20fadd455c0ba88 [id=ec311d46e9224fd38d1864b43717da4a]
I20250111 02:57:51.692714 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_4c9aebb06d554107a5448d968fdb0436 [id=3fab00618e6548de85b5d20adee4e9ac]
I20250111 02:57:51.693554 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_b44f1ebbaf7045e2bcfefcee1c91c0d9 [id=ec5cf5f9b6234008914c0ed664260dcb]
I20250111 02:57:51.695871 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_e091dc5f47d9492aa86ff883e04fe415 [id=ee752760552944289c461f1c4c50d90f]
I20250111 02:57:51.698434 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_1ab2b1f61ced4c97ad5e44f18b6e8699 [id=40eb021e024747f0a173bf87538b7281]
I20250111 02:57:51.700146 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_49e4dd58d29f4cb6a0ed4f85138713b8 [id=f1373922978a4598b888e1863525942e]
I20250111 02:57:51.704398 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_bdacd508498241408a549fdbc6704512 [id=46a029216fdb4496bf9df1d776ccdf7e]
I20250111 02:57:51.708683 20623 catalog_manager.cc:670] Loaded metadata for table default.Table_7ee7d20cfb7e4985bdd8539d15e354fe [id=ff66cd83751a45a09d8012cc8a9f1758]
I20250111 02:57:51.718331 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_a68194fdb909449f878695d09b81c753 [id=49aa9df32f8e4001b66d751832a64534]
I20250111 02:57:51.731245 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_bf50e4c0392544f8a24b861d3edaf46e [id=4b08fa373b0b460094c09b4d3353951c]
I20250111 02:57:51.733106 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_4f96c1244592444a994b5c5631eaaadc [id=4c817a432cf649799a2589f12a320ebd]
I20250111 02:57:51.739018 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_39f0114c91ef4d1c86ef0cf522e31e11 [id=5189b95a9b0e4b47ac0d7768e08e902b]
I20250111 02:57:51.741164 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_dacea9000ff840659052c41c94a8bfaf [id=51bd1567dc9c43b2877791b72c2c2046]
I20250111 02:57:51.760417 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_535081a0b0ff46998f523371377b75f9 [id=531ca04d38e9427eb955326135436b79]
I20250111 02:57:51.853891 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_97e52a18fbd44f189d46ba9b26cd9033 [id=566e89e7552f4ce08fba16fe299b9ccf]
I20250111 02:57:51.876547 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_0dcad07d56d843e59f351d933e625a21 [id=57e820dd99764a578d0c50f5f250aad4]
I20250111 02:57:51.886600 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]
I20250111 02:57:51.906646 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_85a0376bcaf24d0081e780e8c19764b8 [id=5f97e9e0988848319a9c8aaca6aad982]
I20250111 02:57:51.927122 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_c2fa721b96ba4e9ca293ff4b5c455447 [id=64d78652fc6f44778133bd3157964db5]
I20250111 02:57:51.929648 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_f36240205653470e9a67400b787ce860 [id=69f6e7b3ec40423a9d0bef5487ffe84a]
I20250111 02:57:51.930483 20623 tablet_loader.cc:96] loaded metadata for tablet 0158a8849bf44a4fbb5daca22883478f (table default.Table_c2fa721b96ba4e9ca293ff4b5c455447 [id=64d78652fc6f44778133bd3157964db5])
W20250111 02:57:51.943244 18989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 98958.3us
I20250111 02:57:51.960844 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_52d947397a774f96a4fdb5087da4e19a [id=7196ac9c0c4d4dada6c94803e1220caa]
I20250111 02:57:51.961485 20623 tablet_loader.cc:96] loaded metadata for tablet 0255034337f24337be838425136d8b4c (table default.Table_813d4e8bb0f34a46bf96ed3bee6ca86f [id=d790288a097a4416b68b93db6ac847fd])
I20250111 02:57:51.988677 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_4e3a79a5d2184687a8c51fd2702c5fd0 [id=73c59c0d9864497fbc9f19e265d8283f]
I20250111 02:57:51.990756 20676 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:51.994203 20676 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:51.997450 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_b9769d513090402bb596c949defad0df [id=76e72ea89f6e408fbaeda0efc7e79770]
I20250111 02:57:52.005254 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 13 candidate_status { last_received { term: 11 index: 283 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:51.999145 20623 tablet_loader.cc:96] loaded metadata for tablet 07f15096ea484b499fbdecec74eb6f66 (table default.Table_7653ac6289d7467590ac03b43df1b8bf [id=043fd6ad4f5c4a2685af5bbecda90328])
I20250111 02:57:52.026341 20676 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:52.029125 20623 tablet_loader.cc:96] loaded metadata for tablet 0b6687f39d4246b49c0c3eb904b11c1a (table default.Table_857f04d0de7e47a4828db9a8997f1009 [id=9cf908e0c7d44b60b88a32cf2d6368a0])
I20250111 02:57:52.032866 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_6a0ed66485f640e9a2c11b62b5cbd239 [id=7b936eef4300409e908cc92b1453b500]
I20250111 02:57:52.034483 20623 tablet_loader.cc:96] loaded metadata for tablet 0c669a3ff8aa46fa8b04b86ed3bfde49 (table default.Table_fc5930652905494584fde76fff8629d8 [id=a25d2996d4d64d91980bb5a067f5b118])
I20250111 02:57:52.037400 20623 tablet_loader.cc:96] loaded metadata for tablet 0d4ba33836f04ed19d67b8bc161e57c4 (table default.Table_bf5cf724093d4691a60379b87cfab2f0 [id=c151a212f267426a9cb6180c938a5815])
I20250111 02:57:52.052100 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_8eae98e1e49c4f6c9e52644256812eaa [id=7dfae414add74b6491f2f7595a551342]
I20250111 02:57:52.066587 20623 tablet_loader.cc:96] loaded metadata for tablet 12065effb54b4d4a94349641109b578c (table default.Table_d7e00ace61e545129a2e99093dba16db [id=d0e7184505c24fc5b9960e8b6d1c955d])
I20250111 02:57:52.075423 20623 tablet_loader.cc:96] loaded metadata for tablet 16d88c5da4f345c3a43a1ef00f82856b (table default.Table_a7b2b5af5dfb439a84159b75de0d03b1 [id=dc7854580f744d4083f6aa3f0f0f7bf2])
I20250111 02:57:52.081009 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]
I20250111 02:57:52.088804 20623 tablet_loader.cc:96] loaded metadata for tablet 1b69a8c2caea4bddac3f539733bcd3af (table default.Table_bf50e4c0392544f8a24b861d3edaf46e [id=4b08fa373b0b460094c09b4d3353951c])
I20250111 02:57:52.098086 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_53d44be8eaeb4e59a8e3454aac56daca [id=8de61815471842e593a7d29efffc443c]
I20250111 02:57:52.100265 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0]
I20250111 02:57:52.101482 20623 tablet_loader.cc:96] loaded metadata for tablet 1ef92ac716824107916a1ea4fd49d5aa (table default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609])
W20250111 02:57:52.104319 18986 consensus_peers.cc:487] T 5cf1ce65ccf54358b95665b9764fbdb0 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cf1ce65ccf54358b95665b9764fbdb0. This is attempt 11: this message will repeat every 5th retry.
I20250111 02:57:52.105868 20623 tablet_loader.cc:96] loaded metadata for tablet 2009d6f54e2842e8be3af2cc9147dc2c (table default.Table_24e5a7f819cc45dbb7efbd7286ad7005 [id=c04c27fb52054de185e1f3075f89a59a])
I20250111 02:57:52.107972 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_5e7c139097914dd7b7c8e55058d5290d [id=9143d1d3b8754ac78d615a1cc4601d86]
I20250111 02:57:52.119504 20623 tablet_loader.cc:96] loaded metadata for tablet 218b87091d9546ba859f954a8f9e089e (table default.Table_c398ec562f594aca8c1e62cde38c04f9 [id=14bc02cbe30c486782d8fa4396855a9a])
I20250111 02:57:52.149035 20623 tablet_loader.cc:96] loaded metadata for tablet 220da826a0f140d59193e8ad72096b60 (table default.Table_5e7c139097914dd7b7c8e55058d5290d [id=9143d1d3b8754ac78d615a1cc4601d86])
I20250111 02:57:52.150573 20623 tablet_loader.cc:96] loaded metadata for tablet 237ea33846044127987c1d81a3203553 (table default.Table_b9769d513090402bb596c949defad0df [id=76e72ea89f6e408fbaeda0efc7e79770])
I20250111 02:57:52.157042 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605]
I20250111 02:57:52.214619 20623 tablet_loader.cc:96] loaded metadata for tablet 29f27ed2c53d43d7a51ee2e45afaf9fc (table default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7])
I20250111 02:57:52.161302 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_c0ddb0fbf2f347c0b82acf9060722ed2 [id=964e4311f5ed4578a6f7285a9d2bbafa]
I20250111 02:57:52.216964 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_02b5991089a34b0b8ee6039ccc092c93 [id=977f59b134154bc396bf42bc1e44e717]
I20250111 02:57:52.231525 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_98958a28faeb43e8a3b34fcf20d936aa [id=98e24956666a413cbf4564438a594ef0]
I20250111 02:57:52.239449 20623 tablet_loader.cc:96] loaded metadata for tablet 2f11541b436e42b7b0728da2aaec48f7 (table default.Table_f36240205653470e9a67400b787ce860 [id=69f6e7b3ec40423a9d0bef5487ffe84a])
I20250111 02:57:52.251025 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_f9d19027fe814825a6c3906baef77ff3 [id=9a4fbc1c2c2c44c182f9935156885c5a]
I20250111 02:57:52.251675 20623 tablet_loader.cc:96] loaded metadata for tablet 315ac86194b04802909a573876fce64b (table default.Table_ea068446b5434b3ca5f0afa8386c0655 [id=dc09e842add34ef099edfd1296255c4b])
I20250111 02:57:52.252935 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_857f04d0de7e47a4828db9a8997f1009 [id=9cf908e0c7d44b60b88a32cf2d6368a0]
I20250111 02:57:52.254693 20623 tablet_loader.cc:96] loaded metadata for tablet 350f7c7f14004bb4bc3cb2338c7f172f (table default.Table_28d700d6f8444cfeab3dc52dd2bec17f [id=0e5eefce9dfa453eb277f75189def89e])
I20250111 02:57:52.258760 20623 tablet_loader.cc:96] loaded metadata for tablet 3514d795a0f24adcaa2accc0cf05b142 (table default.Table_4c9aebb06d554107a5448d968fdb0436 [id=3fab00618e6548de85b5d20adee4e9ac])
I20250111 02:57:52.262152 20623 tablet_loader.cc:96] loaded metadata for tablet 36f71f8dd2894d40b3bfdd20c1453787 (table default.Table_cce40438ec8e4527b01088af6b7b0981 [id=cb42d3930a9a44bfb5dc40feb979d505])
I20250111 02:57:52.272404 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7]
I20250111 02:57:52.270423 20623 tablet_loader.cc:96] loaded metadata for tablet 3c57d85390324bbc9ea30a709faec01c (table default.Table_52d947397a774f96a4fdb5087da4e19a [id=7196ac9c0c4d4dada6c94803e1220caa])
I20250111 02:57:52.273931 20623 tablet_loader.cc:96] loaded metadata for tablet 3d873a0faf4a4645a3969067ae0583dc (table default.Table_c9582d98645645f3aa80a65f7652cba8 [id=2d5d8f8a7ade443f91a7da34add25a91])
I20250111 02:57:52.274174 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_1431a031eea3496b9764a373a012a367 [id=a0cdf8fe647d4c84a9ce97df9ce9a0ce]
I20250111 02:57:52.286612 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_fc5930652905494584fde76fff8629d8 [id=a25d2996d4d64d91980bb5a067f5b118]
I20250111 02:57:52.288458 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_bdf982e1d6904892ac08ff16ed20a636 [id=a2b6467b58754b85ab0481a9e8aa0ace]
I20250111 02:57:52.290549 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19]
I20250111 02:57:52.293443 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_ee18021e6a544ed6b86b624aec8c39be [id=a9538f81833c4c0d8d02b6271a67f711]
I20250111 02:57:52.290647 20623 tablet_loader.cc:96] loaded metadata for tablet 40fdec982cf643c68ef90798f4f65ad4 (table default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927])
I20250111 02:57:52.295240 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_2c181f66f28741b7b388a7b563b7d2f5 [id=ac0e88a97cf740fd8df3e6cbd81888b5]
I20250111 02:57:52.297102 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3]
I20250111 02:57:52.299358 20623 tablet_loader.cc:96] loaded metadata for tablet 49616e4e7cc647509e72cde57b87d6fa (table default.Table_c4faffc9659848ea9bff581e8d88d40e [id=e709d8f708144de8a18d241f0e0306a0])
I20250111 02:57:52.300794 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_6ad0b827920c42c68113310190019df1 [id=b25314ef99fc4e0ba80c6391223d2a3d]
I20250111 02:57:52.302474 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_b19a186d285d4b5b845727344f7d155b [id=b28b429fe5844cd6ac6f4de349c9449b]
I20250111 02:57:52.304255 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0]
I20250111 02:57:52.305989 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_dfe42fac0a764052a8d2f6add8882b64 [id=b42908f2a2c14d0a84027cb7a1d775b7]
I20250111 02:57:52.309681 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]
I20250111 02:57:52.311682 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_24e5a7f819cc45dbb7efbd7286ad7005 [id=c04c27fb52054de185e1f3075f89a59a]
I20250111 02:57:52.313627 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_23a3f1c40c15458e93c2ef79a3c9e9e2 [id=c12add3740174f66996466994ebe9971]
I20250111 02:57:52.314849 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_bf5cf724093d4691a60379b87cfab2f0 [id=c151a212f267426a9cb6180c938a5815]
I20250111 02:57:52.326054 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_aea3cbca3247424c9a12e310135fe0b9 [id=c151ba8ccc454372b4ad7621b6bc7aeb]
I20250111 02:57:52.347272 20623 tablet_loader.cc:96] loaded metadata for tablet 56fb0bf949904208af003f08caae7594 (table default.Table_23a3f1c40c15458e93c2ef79a3c9e9e2 [id=c12add3740174f66996466994ebe9971])
I20250111 02:57:52.347741 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927]
I20250111 02:57:52.348575 20623 tablet_loader.cc:96] loaded metadata for tablet 5a4bc8e9c0c947f6861a01d3e3bd7f31 (table default.Table_a68194fdb909449f878695d09b81c753 [id=49aa9df32f8e4001b66d751832a64534])
I20250111 02:57:52.349076 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_cce40438ec8e4527b01088af6b7b0981 [id=cb42d3930a9a44bfb5dc40feb979d505]
I20250111 02:57:52.354018 20623 tablet_loader.cc:96] loaded metadata for tablet 63aef98c83d94918bb3163d491870084 (table default.Table_98958a28faeb43e8a3b34fcf20d936aa [id=98e24956666a413cbf4564438a594ef0])
I20250111 02:57:52.355192 20623 tablet_loader.cc:96] loaded metadata for tablet 64abb30da7fc452cad774bbe59b498f4 (table default.Table_26836b21f0f84973b442a4966f97d1ee [id=0037843fe90a422399b837b548ff45b9])
I20250111 02:57:52.356315 20623 tablet_loader.cc:96] loaded metadata for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038])
I20250111 02:57:52.358156 20623 tablet_loader.cc:96] loaded metadata for tablet 6ff06134303d4cb0b4292fe1c0c2d443 (table default.Table_2c181f66f28741b7b388a7b563b7d2f5 [id=ac0e88a97cf740fd8df3e6cbd81888b5])
I20250111 02:57:52.359263 20623 tablet_loader.cc:96] loaded metadata for tablet 7145efe5c0094b70be2cd6481b770248 (table default.Table_7ee7d20cfb7e4985bdd8539d15e354fe [id=ff66cd83751a45a09d8012cc8a9f1758])
I20250111 02:57:52.362323 20623 tablet_loader.cc:96] loaded metadata for tablet 788c34eff1884d979add7e09615c02ae (table default.Table_dfe42fac0a764052a8d2f6add8882b64 [id=b42908f2a2c14d0a84027cb7a1d775b7])
I20250111 02:57:52.364190 20623 tablet_loader.cc:96] loaded metadata for tablet 79f4500d3beb4c9bb20f84c661c3758b (table default.Table_bdacd508498241408a549fdbc6704512 [id=46a029216fdb4496bf9df1d776ccdf7e])
I20250111 02:57:52.365163 20623 tablet_loader.cc:96] loaded metadata for tablet 7c3c5b038d994d07abfea701194e8ad9 (table default.Table_bdf982e1d6904892ac08ff16ed20a636 [id=a2b6467b58754b85ab0481a9e8aa0ace])
I20250111 02:57:52.366578 20623 tablet_loader.cc:96] loaded metadata for tablet 7ea78feee64340e39404cba0f1bf04db (table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5])
I20250111 02:57:52.367403 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5]
I20250111 02:57:52.367827 20623 tablet_loader.cc:96] loaded metadata for tablet 8414653def014ce98bc38d87aac75bae (table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204])
I20250111 02:57:52.368923 20623 tablet_loader.cc:96] loaded metadata for tablet 845da3ee0863430ea3fe9de8c84f2c17 (table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3])
I20250111 02:57:52.370034 20623 tablet_loader.cc:96] loaded metadata for tablet 855dac47872d4c5fbf8f5e38b588c7fe (table default.Table_49e4dd58d29f4cb6a0ed4f85138713b8 [id=f1373922978a4598b888e1863525942e])
I20250111 02:57:52.370163 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_d7e00ace61e545129a2e99093dba16db [id=d0e7184505c24fc5b9960e8b6d1c955d]
I20250111 02:57:52.371600 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3]
I20250111 02:57:52.372579 20623 tablet_loader.cc:96] loaded metadata for tablet 8d20e8099b964fbc9e8e41447772d8d5 (table default.Table_a02982c32dcb409385f8f4e99b95423f [id=354a3a4b0b774fb9bc83623b2cb87f8a])
I20250111 02:57:52.373673 20623 tablet_loader.cc:96] loaded metadata for tablet 8eb87813c28742edad99470c84269ef4 (table default.Table_f9181ca352384593b93119fb3e0cd812 [id=e0ac9d601c464a3daaaa63c4cb90e471])
I20250111 02:57:52.373708 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_6823aef149894f49822fe303c036446d [id=d5ffe760744748a98f80f25b8092f9fa]
I20250111 02:57:52.374686 20623 tablet_loader.cc:96] loaded metadata for tablet 900d4d665c4a4048bc6989ba63391cc1 (table default.Table_baa2b0022e914aa5b485db90037be4af [id=3f31cb2959f340e3ae2656b63eaa52e6])
I20250111 02:57:52.376523 20623 tablet_loader.cc:96] loaded metadata for tablet 9192160e931445f1acd207604192f759 (table default.Table_bccba04b48cc408d845624ee7ed98e5f [id=1496848ec9594cad889312978436d9c4])
I20250111 02:57:52.378351 20623 tablet_loader.cc:96] loaded metadata for tablet 933828fd6fb041e9a4889f4ea8a3c5ff (table default.Table_dc22147bd8ba4ea18bc4f41b25b277c0 [id=1d14d647d8c24e31a84ce6ac7e2c217e])
I20250111 02:57:52.379490 20623 tablet_loader.cc:96] loaded metadata for tablet 934c900baf1b4000af7d7ec60cbb3591 (table default.Table_4f96c1244592444a994b5c5631eaaadc [id=4c817a432cf649799a2589f12a320ebd])
I20250111 02:57:52.381132 20623 tablet_loader.cc:96] loaded metadata for tablet 94bb9089c71b49bea9c0f4d94fc22fb1 (table default.Table_b19a186d285d4b5b845727344f7d155b [id=b28b429fe5844cd6ac6f4de349c9449b])
I20250111 02:57:52.382946 20623 tablet_loader.cc:96] loaded metadata for tablet 9d6de202f4e547a0b8b90cfa09ad319a (table default.Table_6823aef149894f49822fe303c036446d [id=d5ffe760744748a98f80f25b8092f9fa])
I20250111 02:57:52.383926 20623 tablet_loader.cc:96] loaded metadata for tablet 9ee884f4d31f46c197c99bf72f1f19bf (table default.Table_85a0376bcaf24d0081e780e8c19764b8 [id=5f97e9e0988848319a9c8aaca6aad982])
I20250111 02:57:52.386456 20623 tablet_loader.cc:96] loaded metadata for tablet a2eae136623e4802b083d6526c6867c9 (table default.Table_97e52a18fbd44f189d46ba9b26cd9033 [id=566e89e7552f4ce08fba16fe299b9ccf])
I20250111 02:57:52.387648 20623 tablet_loader.cc:96] loaded metadata for tablet a6e4b222bf974a8d80b25141a5686761 (table default.Table_5e21879e67544c71af627f795fd6d940 [id=d6c0153a97cd42d49f534f48fc5b8287])
I20250111 02:57:52.388850 20623 tablet_loader.cc:96] loaded metadata for tablet a983a670c2104398ac1a159ac3a1da7c (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19])
I20250111 02:57:52.389951 20623 tablet_loader.cc:96] loaded metadata for tablet aaab428f798a44f69c80e5792d56b920 (table default.Table_e42719b273bf4b8f96fa48d1a3d6c8d8 [id=27f7d429d88b45b8b76d416b9dc582db])
I20250111 02:57:52.391005 20623 tablet_loader.cc:96] loaded metadata for tablet aca35147034446b3882b5244923bddc7 (table default.Table_1431a031eea3496b9764a373a012a367 [id=a0cdf8fe647d4c84a9ce97df9ce9a0ce])
I20250111 02:57:52.392117 20623 tablet_loader.cc:96] loaded metadata for tablet adf61f6cb63d4e8ea46b440e7ecce00d (table default.Table_c0ddb0fbf2f347c0b82acf9060722ed2 [id=964e4311f5ed4578a6f7285a9d2bbafa])
I20250111 02:57:52.393769 20623 tablet_loader.cc:96] loaded metadata for tablet aec6dffdaa6b4798b7827a6a445601ca (table default.Table_0dcad07d56d843e59f351d933e625a21 [id=57e820dd99764a578d0c50f5f250aad4])
I20250111 02:57:52.395566 20623 tablet_loader.cc:96] loaded metadata for tablet b2f561cfd4f34426b7e42027e53b3ff1 (table default.Table_37a329a4875c463ea093ad5ab23e9cf6 [id=23a0f0c1aaa149c68390a0d0327ff2c5])
I20250111 02:57:52.374922 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_5e21879e67544c71af627f795fd6d940 [id=d6c0153a97cd42d49f534f48fc5b8287]
I20250111 02:57:52.410091 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_813d4e8bb0f34a46bf96ed3bee6ca86f [id=d790288a097a4416b68b93db6ac847fd]
I20250111 02:57:52.412432 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_23d632d4476c46b78d27619201589641 [id=db5ffbbe4fe7424f943fbaca17103d84]
I20250111 02:57:52.413763 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_ea068446b5434b3ca5f0afa8386c0655 [id=dc09e842add34ef099edfd1296255c4b]
I20250111 02:57:52.396675 20623 tablet_loader.cc:96] loaded metadata for tablet b4273afc264843029ed0baf1fc5c8a3e (table default.Table_4e3a79a5d2184687a8c51fd2702c5fd0 [id=73c59c0d9864497fbc9f19e265d8283f])
I20250111 02:57:52.435122 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_a7b2b5af5dfb439a84159b75de0d03b1 [id=dc7854580f744d4083f6aa3f0f0f7bf2]
I20250111 02:57:52.436331 20623 tablet_loader.cc:96] loaded metadata for tablet b5148fa2e6504561bd1b1a19e48993d7 (table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0])
I20250111 02:57:52.437294 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_f9181ca352384593b93119fb3e0cd812 [id=e0ac9d601c464a3daaaa63c4cb90e471]
I20250111 02:57:52.438551 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_c4faffc9659848ea9bff581e8d88d40e [id=e709d8f708144de8a18d241f0e0306a0]
I20250111 02:57:52.441849 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_799d0b8328de40a7bacb58ea433fec72 [id=e9efdc6c94574ddca092dd6b9f3b8419]
I20250111 02:57:52.445678 20623 tablet_loader.cc:96] loaded metadata for tablet bc0bd71c6f4d4c90a4f3ea747af02165 (table default.Table_535081a0b0ff46998f523371377b75f9 [id=531ca04d38e9427eb955326135436b79])
I20250111 02:57:52.447449 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_902d917b19634ad5b20fadd455c0ba88 [id=ec311d46e9224fd38d1864b43717da4a]
I20250111 02:57:52.448880 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_b44f1ebbaf7045e2bcfefcee1c91c0d9 [id=ec5cf5f9b6234008914c0ed664260dcb]
I20250111 02:57:52.450968 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_e091dc5f47d9492aa86ff883e04fe415 [id=ee752760552944289c461f1c4c50d90f]
I20250111 02:57:52.450944 20623 tablet_loader.cc:96] loaded metadata for tablet c8f0c42e06d4428284fa006fe0062cf0 (table default.Table_6ad0b827920c42c68113310190019df1 [id=b25314ef99fc4e0ba80c6391223d2a3d])
I20250111 02:57:52.452181 20623 tablet_loader.cc:96] loaded metadata for tablet cb96635f30b34de48870daf5166813bc (table default.Table_39f0114c91ef4d1c86ef0cf522e31e11 [id=5189b95a9b0e4b47ac0d7768e08e902b])
I20250111 02:57:52.452297 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_49e4dd58d29f4cb6a0ed4f85138713b8 [id=f1373922978a4598b888e1863525942e]
I20250111 02:57:52.453388 20623 tablet_loader.cc:96] loaded metadata for tablet cc7158a7fc624fa4bc816ac3721c610d (table default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605])
I20250111 02:57:52.454548 20623 tablet_loader.cc:96] loaded metadata for tablet d2fc07e0474e4fc5a46f2350229ca31f (table default.Table_6a0ed66485f640e9a2c11b62b5cbd239 [id=7b936eef4300409e908cc92b1453b500])
I20250111 02:57:52.455660 20623 tablet_loader.cc:96] loaded metadata for tablet d3b7616fe56f4e5fb49bdde6961fa008 (table default.Table_1ab2b1f61ced4c97ad5e44f18b6e8699 [id=40eb021e024747f0a173bf87538b7281])
I20250111 02:57:52.456736 20623 tablet_loader.cc:96] loaded metadata for tablet d6da575f052d4b20a8e41f0bd9dab7e4 (table default.Table_02b5991089a34b0b8ee6039ccc092c93 [id=977f59b134154bc396bf42bc1e44e717])
I20250111 02:57:52.458526 20623 tablet_loader.cc:96] loaded metadata for tablet d837bd001aa649e9b3ca94f7cdd9f0fe (table default.Table_b6421db891c148c688d25610559fe16f [id=190cfd157c8d4bd19a8f40cfd80925c1])
I20250111 02:57:52.459656 20623 tablet_loader.cc:96] loaded metadata for tablet d83cfd85a52f438dabe49b6223d5090f (table default.Table_799d0b8328de40a7bacb58ea433fec72 [id=e9efdc6c94574ddca092dd6b9f3b8419])
I20250111 02:57:52.461218 20623 tablet_loader.cc:96] loaded metadata for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe])
I20250111 02:57:52.462468 20623 tablet_loader.cc:96] loaded metadata for tablet dea4db154779483cb069118e3f3789b0 (table default.Table_8eae98e1e49c4f6c9e52644256812eaa [id=7dfae414add74b6491f2f7595a551342])
I20250111 02:57:52.463550 20623 tablet_loader.cc:96] loaded metadata for tablet df1870cceafe463b8d024e8b065850e0 (table default.Table_aea3cbca3247424c9a12e310135fe0b9 [id=c151ba8ccc454372b4ad7621b6bc7aeb])
I20250111 02:57:52.464634 20623 tablet_loader.cc:96] loaded metadata for tablet e01771c56ea5498a876d3bb865c752a2 (table default.Table_dacea9000ff840659052c41c94a8bfaf [id=51bd1567dc9c43b2877791b72c2c2046])
I20250111 02:57:52.465709 20623 tablet_loader.cc:96] loaded metadata for tablet e46894710a72407993b36801f3b90eb0 (table default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3])
I20250111 02:57:52.466759 20623 tablet_loader.cc:96] loaded metadata for tablet e4bd42ad66064bcaa043746081f1c759 (table default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7])
I20250111 02:57:52.467932 20623 tablet_loader.cc:96] loaded metadata for tablet ea76b354a75041e888cf862250e4947b (table default.Table_b44f1ebbaf7045e2bcfefcee1c91c0d9 [id=ec5cf5f9b6234008914c0ed664260dcb])
I20250111 02:57:52.471264 20623 tablet_loader.cc:96] loaded metadata for tablet f1802e7349934fbb9d5f7eda37c3df20 (table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0])
I20250111 02:57:52.472359 20623 tablet_loader.cc:96] loaded metadata for tablet f31ad2e1895d4a26b8e9f7cd7366ef81 (table default.Table_23d632d4476c46b78d27619201589641 [id=db5ffbbe4fe7424f943fbaca17103d84])
I20250111 02:57:52.474723 20623 tablet_loader.cc:96] loaded metadata for tablet f778763e59a94969b3f62be87e2fd867 (table default.Table_902d917b19634ad5b20fadd455c0ba88 [id=ec311d46e9224fd38d1864b43717da4a])
I20250111 02:57:52.475700 20623 tablet_loader.cc:96] loaded metadata for tablet f9092a3453f94abbbf4bdd66a57d42cc (table default.Table_ee18021e6a544ed6b86b624aec8c39be [id=a9538f81833c4c0d8d02b6271a67f711])
W20250111 02:57:52.475754 19133 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.65:34535, user_credentials={real_user=slave}} blocked reactor thread for 110413us
I20250111 02:57:52.477483 20623 tablet_loader.cc:96] loaded metadata for tablet f9ec6b2a27744ee1aef0efdc98d8757c (table default.Table_f9d19027fe814825a6c3906baef77ff3 [id=9a4fbc1c2c2c44c182f9935156885c5a])
I20250111 02:57:52.478428 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_7ee7d20cfb7e4985bdd8539d15e354fe [id=ff66cd83751a45a09d8012cc8a9f1758]
I20250111 02:57:52.480738 20623 tablet_loader.cc:96] loaded metadata for tablet fa86b1e6eca7403c9a9c52592b8fb636 (table default.Table_4b9e85e01e1843ce831205177cbb2e0f [id=0a118da8c5d0408980cbd0b33e0c2ed0])
I20250111 02:57:52.481832 20623 tablet_loader.cc:96] loaded metadata for tablet fb9b1c7e252d40e29e6d16311d145968 (table default.Table_e091dc5f47d9492aa86ff883e04fe415 [id=ee752760552944289c461f1c4c50d90f])
I20250111 02:57:52.482951 20623 tablet_loader.cc:96] loaded metadata for tablet ffd063f8889b4bcdb9656466f0b144f3 (table default.Table_53d44be8eaeb4e59a8e3454aac56daca [id=8de61815471842e593a7d29efffc443c])
W20250111 02:57:52.513689 20623 catalog_manager.cc:1477] Time spent T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c: Loading table and tablet metadata into memory: real 3.910s user 0.362s sys 0.053s
I20250111 02:57:52.514071 20623 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:57:52.514761 18717 raft_consensus.cc:1270] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 12 FOLLOWER]: Refusing update from remote peer 8784b9603f8544b79373158aace063af: Log matching property violated. Preceding OpId in replica: term: 11 index: 283. Preceding OpId from leader: term: 12 index: 283. (term mismatch)
I20250111 02:57:52.515215 18717 pending_rounds.cc:77] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c: Aborting all ops after (but not including) 282
I20250111 02:57:52.515704 18717 pending_rounds.cc:99] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c: Aborting uncommitted NO_OP operation due to leader change: 11.283
I20250111 02:57:52.516101 18717 raft_consensus.cc:2883] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250111 02:57:52.517005 18717 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36846 (request call id 348) took 2957 ms. Trace:
I20250111 02:57:52.517465 18717 rpcz_store.cc:276] 0111 02:57:49.559346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:49.561131 (+ 1785us) service_pool.cc:225] Handling call
0111 02:57:52.513915 (+2952784us) raft_consensus.cc:1466] Updating replica for 0 ops
0111 02:57:52.516674 (+ 2759us) raft_consensus.cc:1692] Filling consensus response to leader.
0111 02:57:52.516938 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:52.515785 18712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 12 candidate_status { last_received { term: 10 index: 282 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:52.520449 20654 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.001s
I20250111 02:57:52.538914 18712 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 12.
W20250111 02:57:52.539611 18712 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36846 (request call id 347) took 2982 ms (client timeout 729 ms). Trace:
W20250111 02:57:52.540071 18712 rpcz_store.cc:269] 0111 02:57:49.557691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:49.560827 (+ 3136us) service_pool.cc:225] Handling call
0111 02:57:52.539529 (+2978702us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:52.519081 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 12 candidate_status { last_received { term: 10 index: 282 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:52.541514 18713 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 12.
W20250111 02:57:52.542110 18713 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36846 (request call id 345) took 3859 ms (client timeout 729 ms). Trace:
W20250111 02:57:52.542526 18713 rpcz_store.cc:269] 0111 02:57:48.682548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:48.682704 (+ 156us) service_pool.cc:225] Handling call
0111 02:57:52.542042 (+3859338us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:52.519989 18718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 12 candidate_status { last_received { term: 10 index: 282 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c"
I20250111 02:57:52.520872 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 13 candidate_status { last_received { term: 12 index: 283 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:52.547488 20070 tablet_loader.cc:96] loaded metadata for tablet 0158a8849bf44a4fbb5daca22883478f (table default.Table_c2fa721b96ba4e9ca293ff4b5c455447 [id=64d78652fc6f44778133bd3157964db5])
W20250111 02:57:52.547597 18719 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46906 (request call id 15) took 2451 ms (client timeout 607 ms). Trace:
W20250111 02:57:52.547976 18719 rpcz_store.cc:269] 0111 02:57:50.096153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:50.096362 (+ 209us) service_pool.cc:225] Handling call
0111 02:57:52.547526 (+2451164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:52.549083 20070 tablet_loader.cc:96] loaded metadata for tablet 0255034337f24337be838425136d8b4c (table default.Table_813d4e8bb0f34a46bf96ed3bee6ca86f [id=d790288a097a4416b68b93db6ac847fd])
I20250111 02:57:52.516270 18714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 13 candidate_status { last_received { term: 12 index: 283 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:52.537324 20623 catalog_manager.cc:1260] Loaded cluster ID: 505f17df54394056975cd03f4ae103e8
I20250111 02:57:52.549746 20623 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
W20250111 02:57:52.550105 18714 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46906 (request call id 16) took 1785 ms (client timeout 607 ms). Trace:
W20250111 02:57:52.550529 18714 rpcz_store.cc:269] 0111 02:57:50.764860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:50.765184 (+ 324us) service_pool.cc:225] Handling call
0111 02:57:52.550031 (+1784847us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:52.516571 18716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8784b9603f8544b79373158aace063af" candidate_term: 12 candidate_status { last_received { term: 10 index: 282 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c"
W20250111 02:57:52.551873 18716 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36846 (request call id 349) took 2934 ms (client timeout 695 ms). Trace:
W20250111 02:57:52.552265 18716 rpcz_store.cc:269] 0111 02:57:49.617777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:49.618005 (+ 228us) service_pool.cc:225] Handling call
0111 02:57:52.551786 (+2933781us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:52.566218 20070 tablet_loader.cc:96] loaded metadata for tablet 07f15096ea484b499fbdecec74eb6f66 (table default.Table_7653ac6289d7467590ac03b43df1b8bf [id=043fd6ad4f5c4a2685af5bbecda90328])
I20250111 02:57:52.575649 20623 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:57:52.583209 20070 tablet_loader.cc:96] loaded metadata for tablet 0b6687f39d4246b49c0c3eb904b11c1a (table default.Table_857f04d0de7e47a4828db9a8997f1009 [id=9cf908e0c7d44b60b88a32cf2d6368a0])
I20250111 02:57:52.584614 20070 tablet_loader.cc:96] loaded metadata for tablet 0c669a3ff8aa46fa8b04b86ed3bfde49 (table default.Table_fc5930652905494584fde76fff8629d8 [id=a25d2996d4d64d91980bb5a067f5b118])
I20250111 02:57:52.585834 20070 tablet_loader.cc:96] loaded metadata for tablet 0d4ba33836f04ed19d67b8bc161e57c4 (table default.Table_bf5cf724093d4691a60379b87cfab2f0 [id=c151a212f267426a9cb6180c938a5815])
I20250111 02:57:52.589545 20070 tablet_loader.cc:96] loaded metadata for tablet 12065effb54b4d4a94349641109b578c (table default.Table_d7e00ace61e545129a2e99093dba16db [id=d0e7184505c24fc5b9960e8b6d1c955d])
I20250111 02:57:52.592059 20070 tablet_loader.cc:96] loaded metadata for tablet 16d88c5da4f345c3a43a1ef00f82856b (table default.Table_a7b2b5af5dfb439a84159b75de0d03b1 [id=dc7854580f744d4083f6aa3f0f0f7bf2])
I20250111 02:57:52.594739 20070 tablet_loader.cc:96] loaded metadata for tablet 1b69a8c2caea4bddac3f539733bcd3af (table default.Table_bf50e4c0392544f8a24b861d3edaf46e [id=4b08fa373b0b460094c09b4d3353951c])
I20250111 02:57:52.596764 20070 tablet_loader.cc:96] loaded metadata for tablet 1ef92ac716824107916a1ea4fd49d5aa (table default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609])
I20250111 02:57:52.603646 20070 tablet_loader.cc:96] loaded metadata for tablet 2009d6f54e2842e8be3af2cc9147dc2c (table default.Table_24e5a7f819cc45dbb7efbd7286ad7005 [id=c04c27fb52054de185e1f3075f89a59a])
I20250111 02:57:52.604794 20070 tablet_loader.cc:96] loaded metadata for tablet 218b87091d9546ba859f954a8f9e089e (table default.Table_c398ec562f594aca8c1e62cde38c04f9 [id=14bc02cbe30c486782d8fa4396855a9a])
I20250111 02:57:52.601261 18718 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8784b9603f8544b79373158aace063af in term 12.
I20250111 02:57:52.606020 20070 tablet_loader.cc:96] loaded metadata for tablet 220da826a0f140d59193e8ad72096b60 (table default.Table_5e7c139097914dd7b7c8e55058d5290d [id=9143d1d3b8754ac78d615a1cc4601d86])
I20250111 02:57:52.607306 20070 tablet_loader.cc:96] loaded metadata for tablet 237ea33846044127987c1d81a3203553 (table default.Table_b9769d513090402bb596c949defad0df [id=76e72ea89f6e408fbaeda0efc7e79770])
I20250111 02:57:52.609875 20070 tablet_loader.cc:96] loaded metadata for tablet 29f27ed2c53d43d7a51ee2e45afaf9fc (table default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7])
I20250111 02:57:52.613240 20070 tablet_loader.cc:96] loaded metadata for tablet 2f11541b436e42b7b0728da2aaec48f7 (table default.Table_f36240205653470e9a67400b787ce860 [id=69f6e7b3ec40423a9d0bef5487ffe84a])
I20250111 02:57:52.622922 20070 tablet_loader.cc:96] loaded metadata for tablet 315ac86194b04802909a573876fce64b (table default.Table_ea068446b5434b3ca5f0afa8386c0655 [id=dc09e842add34ef099edfd1296255c4b])
I20250111 02:57:52.625061 20070 tablet_loader.cc:96] loaded metadata for tablet 350f7c7f14004bb4bc3cb2338c7f172f (table default.Table_28d700d6f8444cfeab3dc52dd2bec17f [id=0e5eefce9dfa453eb277f75189def89e])
I20250111 02:57:52.622781 20623 catalog_manager.cc:5910] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c: Loaded TSK: 0
I20250111 02:57:52.627645 20623 catalog_manager.cc:1515] Initializing in-progress tserver states...
W20250111 02:57:52.621105 18718 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36846 (request call id 346) took 3704 ms (client timeout 695 ms). Trace:
W20250111 02:57:52.631290 18718 rpcz_store.cc:269] 0111 02:57:48.913239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:48.913474 (+ 235us) service_pool.cc:225] Handling call
0111 02:57:52.617910 (+3704436us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:52.636086 20070 tablet_loader.cc:96] loaded metadata for tablet 3514d795a0f24adcaa2accc0cf05b142 (table default.Table_4c9aebb06d554107a5448d968fdb0436 [id=3fab00618e6548de85b5d20adee4e9ac])
I20250111 02:57:52.637615 20070 tablet_loader.cc:96] loaded metadata for tablet 36f71f8dd2894d40b3bfdd20c1453787 (table default.Table_cce40438ec8e4527b01088af6b7b0981 [id=cb42d3930a9a44bfb5dc40feb979d505])
I20250111 02:57:52.644695 20070 tablet_loader.cc:96] loaded metadata for tablet 3c57d85390324bbc9ea30a709faec01c (table default.Table_52d947397a774f96a4fdb5087da4e19a [id=7196ac9c0c4d4dada6c94803e1220caa])
I20250111 02:57:52.645848 20070 tablet_loader.cc:96] loaded metadata for tablet 3d873a0faf4a4645a3969067ae0583dc (table default.Table_c9582d98645645f3aa80a65f7652cba8 [id=2d5d8f8a7ade443f91a7da34add25a91])
I20250111 02:57:52.647931 20070 tablet_loader.cc:96] loaded metadata for tablet 40fdec982cf643c68ef90798f4f65ad4 (table default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927])
I20250111 02:57:52.650610 20070 tablet_loader.cc:96] loaded metadata for tablet 49616e4e7cc647509e72cde57b87d6fa (table default.Table_c4faffc9659848ea9bff581e8d88d40e [id=e709d8f708144de8a18d241f0e0306a0])
I20250111 02:57:52.663384 20070 tablet_loader.cc:96] loaded metadata for tablet 56fb0bf949904208af003f08caae7594 (table default.Table_23a3f1c40c15458e93c2ef79a3c9e9e2 [id=c12add3740174f66996466994ebe9971])
I20250111 02:57:52.664731 20070 tablet_loader.cc:96] loaded metadata for tablet 5a4bc8e9c0c947f6861a01d3e3bd7f31 (table default.Table_a68194fdb909449f878695d09b81c753 [id=49aa9df32f8e4001b66d751832a64534])
I20250111 02:57:52.672343 20070 tablet_loader.cc:96] loaded metadata for tablet 63aef98c83d94918bb3163d491870084 (table default.Table_98958a28faeb43e8a3b34fcf20d936aa [id=98e24956666a413cbf4564438a594ef0])
I20250111 02:57:52.673591 20070 tablet_loader.cc:96] loaded metadata for tablet 64abb30da7fc452cad774bbe59b498f4 (table default.Table_26836b21f0f84973b442a4966f97d1ee [id=0037843fe90a422399b837b548ff45b9])
I20250111 02:57:52.674765 20070 tablet_loader.cc:96] loaded metadata for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038])
I20250111 02:57:52.676765 20070 tablet_loader.cc:96] loaded metadata for tablet 6ff06134303d4cb0b4292fe1c0c2d443 (table default.Table_2c181f66f28741b7b388a7b563b7d2f5 [id=ac0e88a97cf740fd8df3e6cbd81888b5])
I20250111 02:57:52.679261 20070 tablet_loader.cc:96] loaded metadata for tablet 7145efe5c0094b70be2cd6481b770248 (table default.Table_7ee7d20cfb7e4985bdd8539d15e354fe [id=ff66cd83751a45a09d8012cc8a9f1758])
I20250111 02:57:52.683348 20070 tablet_loader.cc:96] loaded metadata for tablet 788c34eff1884d979add7e09615c02ae (table default.Table_dfe42fac0a764052a8d2f6add8882b64 [id=b42908f2a2c14d0a84027cb7a1d775b7])
I20250111 02:57:52.685235 20070 tablet_loader.cc:96] loaded metadata for tablet 79f4500d3beb4c9bb20f84c661c3758b (table default.Table_bdacd508498241408a549fdbc6704512 [id=46a029216fdb4496bf9df1d776ccdf7e])
I20250111 02:57:52.686386 20070 tablet_loader.cc:96] loaded metadata for tablet 7c3c5b038d994d07abfea701194e8ad9 (table default.Table_bdf982e1d6904892ac08ff16ed20a636 [id=a2b6467b58754b85ab0481a9e8aa0ace])
I20250111 02:57:52.687559 20070 tablet_loader.cc:96] loaded metadata for tablet 7ea78feee64340e39404cba0f1bf04db (table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5])
I20250111 02:57:52.688686 20070 tablet_loader.cc:96] loaded metadata for tablet 8414653def014ce98bc38d87aac75bae (table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204])
I20250111 02:57:52.689889 20070 tablet_loader.cc:96] loaded metadata for tablet 845da3ee0863430ea3fe9de8c84f2c17 (table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3])
I20250111 02:57:52.691031 20070 tablet_loader.cc:96] loaded metadata for tablet 855dac47872d4c5fbf8f5e38b588c7fe (table default.Table_49e4dd58d29f4cb6a0ed4f85138713b8 [id=f1373922978a4598b888e1863525942e])
I20250111 02:57:52.687126 20679 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8784b9603f8544b79373158aace063af. Latest consensus state: current_term: 12 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:52.692589 20679 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:52.693867 20070 tablet_loader.cc:96] loaded metadata for tablet 8d20e8099b964fbc9e8e41447772d8d5 (table default.Table_a02982c32dcb409385f8f4e99b95423f [id=354a3a4b0b774fb9bc83623b2cb87f8a])
I20250111 02:57:52.668025 20682 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 12 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:52.705478 20682 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:57:52.719985 20070 tablet_loader.cc:96] loaded metadata for tablet 8eb87813c28742edad99470c84269ef4 (table default.Table_f9181ca352384593b93119fb3e0cd812 [id=e0ac9d601c464a3daaaa63c4cb90e471])
I20250111 02:57:52.721135 20070 tablet_loader.cc:96] loaded metadata for tablet 900d4d665c4a4048bc6989ba63391cc1 (table default.Table_baa2b0022e914aa5b485db90037be4af [id=3f31cb2959f340e3ae2656b63eaa52e6])
I20250111 02:57:52.751224 20070 tablet_loader.cc:96] loaded metadata for tablet 9192160e931445f1acd207604192f759 (table default.Table_bccba04b48cc408d845624ee7ed98e5f [id=1496848ec9594cad889312978436d9c4])
I20250111 02:57:52.753285 20070 tablet_loader.cc:96] loaded metadata for tablet 933828fd6fb041e9a4889f4ea8a3c5ff (table default.Table_dc22147bd8ba4ea18bc4f41b25b277c0 [id=1d14d647d8c24e31a84ce6ac7e2c217e])
I20250111 02:57:52.757970 20070 tablet_loader.cc:96] loaded metadata for tablet 934c900baf1b4000af7d7ec60cbb3591 (table default.Table_4f96c1244592444a994b5c5631eaaadc [id=4c817a432cf649799a2589f12a320ebd])
I20250111 02:57:52.759908 20070 tablet_loader.cc:96] loaded metadata for tablet 94bb9089c71b49bea9c0f4d94fc22fb1 (table default.Table_b19a186d285d4b5b845727344f7d155b [id=b28b429fe5844cd6ac6f4de349c9449b])
I20250111 02:57:52.768086 20070 tablet_loader.cc:96] loaded metadata for tablet 9d6de202f4e547a0b8b90cfa09ad319a (table default.Table_6823aef149894f49822fe303c036446d [id=d5ffe760744748a98f80f25b8092f9fa])
I20250111 02:57:52.769186 20070 tablet_loader.cc:96] loaded metadata for tablet 9ee884f4d31f46c197c99bf72f1f19bf (table default.Table_85a0376bcaf24d0081e780e8c19764b8 [id=5f97e9e0988848319a9c8aaca6aad982])
I20250111 02:57:52.778409 20070 tablet_loader.cc:96] loaded metadata for tablet a2eae136623e4802b083d6526c6867c9 (table default.Table_97e52a18fbd44f189d46ba9b26cd9033 [id=566e89e7552f4ce08fba16fe299b9ccf])
I20250111 02:57:52.779790 20070 tablet_loader.cc:96] loaded metadata for tablet a6e4b222bf974a8d80b25141a5686761 (table default.Table_5e21879e67544c71af627f795fd6d940 [id=d6c0153a97cd42d49f534f48fc5b8287])
I20250111 02:57:52.781080 20070 tablet_loader.cc:96] loaded metadata for tablet a983a670c2104398ac1a159ac3a1da7c (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19])
I20250111 02:57:52.782330 20070 tablet_loader.cc:96] loaded metadata for tablet aaab428f798a44f69c80e5792d56b920 (table default.Table_e42719b273bf4b8f96fa48d1a3d6c8d8 [id=27f7d429d88b45b8b76d416b9dc582db])
I20250111 02:57:52.783496 20070 tablet_loader.cc:96] loaded metadata for tablet aca35147034446b3882b5244923bddc7 (table default.Table_1431a031eea3496b9764a373a012a367 [id=a0cdf8fe647d4c84a9ce97df9ce9a0ce])
I20250111 02:57:52.784631 20070 tablet_loader.cc:96] loaded metadata for tablet adf61f6cb63d4e8ea46b440e7ecce00d (table default.Table_c0ddb0fbf2f347c0b82acf9060722ed2 [id=964e4311f5ed4578a6f7285a9d2bbafa])
I20250111 02:57:52.797825 20070 tablet_loader.cc:96] loaded metadata for tablet aec6dffdaa6b4798b7827a6a445601ca (table default.Table_0dcad07d56d843e59f351d933e625a21 [id=57e820dd99764a578d0c50f5f250aad4])
I20250111 02:57:52.799829 20070 tablet_loader.cc:96] loaded metadata for tablet b2f561cfd4f34426b7e42027e53b3ff1 (table default.Table_37a329a4875c463ea093ad5ab23e9cf6 [id=23a0f0c1aaa149c68390a0d0327ff2c5])
I20250111 02:57:52.800973 20070 tablet_loader.cc:96] loaded metadata for tablet b4273afc264843029ed0baf1fc5c8a3e (table default.Table_4e3a79a5d2184687a8c51fd2702c5fd0 [id=73c59c0d9864497fbc9f19e265d8283f])
I20250111 02:57:52.802124 20070 tablet_loader.cc:96] loaded metadata for tablet b5148fa2e6504561bd1b1a19e48993d7 (table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0])
I20250111 02:57:52.806074 20070 tablet_loader.cc:96] loaded metadata for tablet bc0bd71c6f4d4c90a4f3ea747af02165 (table default.Table_535081a0b0ff46998f523371377b75f9 [id=531ca04d38e9427eb955326135436b79])
I20250111 02:57:52.814481 20070 tablet_loader.cc:96] loaded metadata for tablet c8f0c42e06d4428284fa006fe0062cf0 (table default.Table_6ad0b827920c42c68113310190019df1 [id=b25314ef99fc4e0ba80c6391223d2a3d])
I20250111 02:57:52.815797 20070 tablet_loader.cc:96] loaded metadata for tablet cb96635f30b34de48870daf5166813bc (table default.Table_39f0114c91ef4d1c86ef0cf522e31e11 [id=5189b95a9b0e4b47ac0d7768e08e902b])
W20250111 02:57:52.841349 19095 debug-util.cc:398] Leaking SignalData structure 0x7b08000dc1a0 after lost signal to thread 18966
W20250111 02:57:52.842629 19095 debug-util.cc:398] Leaking SignalData structure 0x7b0800162500 after lost signal to thread 19098
I20250111 02:57:52.816948 20070 tablet_loader.cc:96] loaded metadata for tablet cc7158a7fc624fa4bc816ac3721c610d (table default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605])
I20250111 02:57:52.844156 20070 tablet_loader.cc:96] loaded metadata for tablet d2fc07e0474e4fc5a46f2350229ca31f (table default.Table_6a0ed66485f640e9a2c11b62b5cbd239 [id=7b936eef4300409e908cc92b1453b500])
I20250111 02:57:52.845634 20070 tablet_loader.cc:96] loaded metadata for tablet d3b7616fe56f4e5fb49bdde6961fa008 (table default.Table_1ab2b1f61ced4c97ad5e44f18b6e8699 [id=40eb021e024747f0a173bf87538b7281])
I20250111 02:57:52.846963 20070 tablet_loader.cc:96] loaded metadata for tablet d6da575f052d4b20a8e41f0bd9dab7e4 (table default.Table_02b5991089a34b0b8ee6039ccc092c93 [id=977f59b134154bc396bf42bc1e44e717])
I20250111 02:57:52.849141 20070 tablet_loader.cc:96] loaded metadata for tablet d837bd001aa649e9b3ca94f7cdd9f0fe (table default.Table_b6421db891c148c688d25610559fe16f [id=190cfd157c8d4bd19a8f40cfd80925c1])
I20250111 02:57:52.850427 20070 tablet_loader.cc:96] loaded metadata for tablet d83cfd85a52f438dabe49b6223d5090f (table default.Table_799d0b8328de40a7bacb58ea433fec72 [id=e9efdc6c94574ddca092dd6b9f3b8419])
I20250111 02:57:52.851883 20070 tablet_loader.cc:96] loaded metadata for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe])
I20250111 02:57:52.852887 20070 tablet_loader.cc:96] loaded metadata for tablet dea4db154779483cb069118e3f3789b0 (table default.Table_8eae98e1e49c4f6c9e52644256812eaa [id=7dfae414add74b6491f2f7595a551342])
I20250111 02:57:52.854202 20070 tablet_loader.cc:96] loaded metadata for tablet df1870cceafe463b8d024e8b065850e0 (table default.Table_aea3cbca3247424c9a12e310135fe0b9 [id=c151ba8ccc454372b4ad7621b6bc7aeb])
I20250111 02:57:52.855579 20070 tablet_loader.cc:96] loaded metadata for tablet e01771c56ea5498a876d3bb865c752a2 (table default.Table_dacea9000ff840659052c41c94a8bfaf [id=51bd1567dc9c43b2877791b72c2c2046])
I20250111 02:57:52.857036 20070 tablet_loader.cc:96] loaded metadata for tablet e46894710a72407993b36801f3b90eb0 (table default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3])
I20250111 02:57:52.858355 20070 tablet_loader.cc:96] loaded metadata for tablet e4bd42ad66064bcaa043746081f1c759 (table default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7])
I20250111 02:57:52.860961 20070 tablet_loader.cc:96] loaded metadata for tablet ea76b354a75041e888cf862250e4947b (table default.Table_b44f1ebbaf7045e2bcfefcee1c91c0d9 [id=ec5cf5f9b6234008914c0ed664260dcb])
I20250111 02:57:52.864450 20070 tablet_loader.cc:96] loaded metadata for tablet f1802e7349934fbb9d5f7eda37c3df20 (table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0])
I20250111 02:57:52.865772 20070 tablet_loader.cc:96] loaded metadata for tablet f31ad2e1895d4a26b8e9f7cd7366ef81 (table default.Table_23d632d4476c46b78d27619201589641 [id=db5ffbbe4fe7424f943fbaca17103d84])
I20250111 02:57:52.868518 20070 tablet_loader.cc:96] loaded metadata for tablet f778763e59a94969b3f62be87e2fd867 (table default.Table_902d917b19634ad5b20fadd455c0ba88 [id=ec311d46e9224fd38d1864b43717da4a])
I20250111 02:57:52.869758 20070 tablet_loader.cc:96] loaded metadata for tablet f9092a3453f94abbbf4bdd66a57d42cc (table default.Table_ee18021e6a544ed6b86b624aec8c39be [id=a9538f81833c4c0d8d02b6271a67f711])
I20250111 02:57:52.870893 20070 tablet_loader.cc:96] loaded metadata for tablet f9ec6b2a27744ee1aef0efdc98d8757c (table default.Table_f9d19027fe814825a6c3906baef77ff3 [id=9a4fbc1c2c2c44c182f9935156885c5a])
I20250111 02:57:52.872366 20070 tablet_loader.cc:96] loaded metadata for tablet fa86b1e6eca7403c9a9c52592b8fb636 (table default.Table_4b9e85e01e1843ce831205177cbb2e0f [id=0a118da8c5d0408980cbd0b33e0c2ed0])
I20250111 02:57:52.873500 20070 tablet_loader.cc:96] loaded metadata for tablet fb9b1c7e252d40e29e6d16311d145968 (table default.Table_e091dc5f47d9492aa86ff883e04fe415 [id=ee752760552944289c461f1c4c50d90f])
I20250111 02:57:52.874812 20070 tablet_loader.cc:96] loaded metadata for tablet ffd063f8889b4bcdb9656466f0b144f3 (table default.Table_53d44be8eaeb4e59a8e3454aac56daca [id=8de61815471842e593a7d29efffc443c])
W20250111 02:57:52.919829 20070 catalog_manager.cc:1477] Time spent T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Loading table and tablet metadata into memory: real 16.532s user 0.326s sys 0.112s
I20250111 02:57:52.920267 20070 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:57:52.920315 18631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:52.921104 18639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
W20250111 02:57:52.922061 18639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 23) took 16494 ms (client timeout 697 ms). Trace:
W20250111 02:57:52.922498 18639 rpcz_store.cc:269] 0111 02:57:36.427581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:36.427820 (+ 239us) service_pool.cc:225] Handling call
0111 02:57:52.921940 (+16494120us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:52.922061 18631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 27) took 11891 ms (client timeout 553 ms). Trace:
I20250111 02:57:52.923897 18637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
W20250111 02:57:52.924237 18631 rpcz_store.cc:269] 0111 02:57:41.030583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:41.030821 (+ 238us) service_pool.cc:225] Handling call
0111 02:57:52.921935 (+11891114us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:52.925065 18637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 8) took 15411 ms (client timeout 568 ms). Trace:
W20250111 02:57:52.925518 18637 rpcz_store.cc:269] 0111 02:57:37.514170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:37.514426 (+ 256us) service_pool.cc:225] Handling call
0111 02:57:52.924971 (+15410545us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:52.926103 18639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 13) took 11712 ms (client timeout 588 ms). Trace:
W20250111 02:57:52.926430 18639 rpcz_store.cc:269] 0111 02:57:41.213796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.925822 (+11712026us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.926023 (+ 201us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.926761 18637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 28) took 11317 ms (client timeout 553 ms). Trace:
W20250111 02:57:52.927276 18639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 14) took 11113 ms (client timeout 588 ms). Trace:
W20250111 02:57:52.927723 18639 rpcz_store.cc:269] 0111 02:57:41.813624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.927045 (+11113421us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.927180 (+ 135us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.928519 18639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 29) took 10017 ms (client timeout 661 ms). Trace:
W20250111 02:57:52.928829 18639 rpcz_store.cc:269] 0111 02:57:42.910988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.928262 (+10017274us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.928467 (+ 205us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250111 02:57:52.929898 18636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:52.930464 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
W20250111 02:57:52.931401 18638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 7) took 16005 ms (client timeout 568 ms). Trace:
W20250111 02:57:52.931361 18636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 25) took 14503 ms (client timeout 539 ms). Trace:
W20250111 02:57:52.931761 18638 rpcz_store.cc:269] 0111 02:57:36.926305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:36.926460 (+ 155us) service_pool.cc:225] Handling call
0111 02:57:52.931310 (+16004850us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:52.931806 18636 rpcz_store.cc:269] 0111 02:57:38.427630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:38.427918 (+ 288us) service_pool.cc:225] Handling call
0111 02:57:52.931241 (+14503323us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:52.932986 18636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 30) took 9345 ms (client timeout 661 ms). Trace:
I20250111 02:57:52.933660 18632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
W20250111 02:57:52.934903 18632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 12) took 13201 ms (client timeout 548 ms). Trace:
I20250111 02:57:52.935132 18626 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:47522 (request call id 295) took 16600 ms. Trace:
I20250111 02:57:52.935446 18626 rpcz_store.cc:276] 0111 02:57:36.334775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:36.395177 (+ 60402us) service_pool.cc:225] Handling call
0111 02:57:52.934687 (+16539510us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:52.935412 18632 rpcz_store.cc:269] 0111 02:57:39.733401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:39.733667 (+ 266us) service_pool.cc:225] Handling call
0111 02:57:52.934805 (+13201138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:52.937250 18634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:52.938766 18633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55106 (request call id 11) took 13326 ms. Trace:
W20250111 02:57:52.939373 18638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 15) took 9741 ms (client timeout 653 ms). Trace:
I20250111 02:57:52.939200 18633 rpcz_store.cc:276] 0111 02:57:39.611443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:39.611639 (+ 196us) service_pool.cc:225] Handling call
0111 02:57:52.938149 (+13326510us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:52.939661 18638 rpcz_store.cc:269] 0111 02:57:43.197955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.939139 (+9741184us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.939290 (+ 151us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.940469 18638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 31) took 7756 ms (client timeout 596 ms). Trace:
W20250111 02:57:52.940659 18633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 17) took 7305 ms (client timeout 521 ms). Trace:
W20250111 02:57:52.940783 18638 rpcz_store.cc:269] 0111 02:57:45.184517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.940258 (+7755741us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.940404 (+ 146us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.942138 18639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 32) took 7076 ms (client timeout 596 ms). Trace:
W20250111 02:57:52.942462 18639 rpcz_store.cc:269] 0111 02:57:45.865413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.941977 (+7076564us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.942085 (+ 108us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.944072 18631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 19) took 6772 ms (client timeout 521 ms). Trace:
I20250111 02:57:52.943882 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
W20250111 02:57:52.944502 18631 rpcz_store.cc:269] 0111 02:57:46.172089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.943840 (+6771751us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.944004 (+ 164us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.945574 18631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 18) took 6806 ms (client timeout 643 ms). Trace:
W20250111 02:57:52.945962 18631 rpcz_store.cc:269] 0111 02:57:46.139393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.945358 (+6805965us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.945517 (+ 159us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.946980 18631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 20) took 6140 ms (client timeout 643 ms). Trace:
W20250111 02:57:52.947412 18631 rpcz_store.cc:269] 0111 02:57:46.807117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.946759 (+6139642us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.946917 (+ 158us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.948423 18631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 21) took 4768 ms (client timeout 598 ms). Trace:
W20250111 02:57:52.945269 18635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 26) took 13868 ms (client timeout 539 ms). Trace:
W20250111 02:57:52.950348 18632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 16) took 9087 ms (client timeout 653 ms). Trace:
W20250111 02:57:52.950754 18632 rpcz_store.cc:269] 0111 02:57:43.862667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.936432 (+9073765us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.950243 (+ 13811us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.951903 18632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 34) took 4727 ms (client timeout 613 ms). Trace:
W20250111 02:57:52.950168 18635 rpcz_store.cc:269] 0111 02:57:39.076740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:39.077004 (+ 264us) service_pool.cc:225] Handling call
0111 02:57:52.945152 (+13868148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:52.952888 18640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 281 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
W20250111 02:57:52.953889 18635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 33) took 4762 ms (client timeout 704 ms). Trace:
W20250111 02:57:52.938650 18634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 9) took 13826 ms (client timeout 548 ms). Trace:
W20250111 02:57:52.953955 18640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 24) took 15874 ms (client timeout 697 ms). Trace:
W20250111 02:57:52.954365 18637 rpcz_store.cc:269] 0111 02:57:41.609167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.926497 (+11317330us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.926698 (+ 201us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.956450 18637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 37) took 3980 ms (client timeout 704 ms). Trace:
W20250111 02:57:52.955758 18639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 36) took 3980 ms (client timeout 613 ms). Trace:
W20250111 02:57:52.956871 18637 rpcz_store.cc:269] 0111 02:57:48.976426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.956237 (+3979811us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.956386 (+ 149us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250111 02:57:52.957687 20070 catalog_manager.cc:1260] Loaded cluster ID: 505f17df54394056975cd03f4ae103e8
W20250111 02:57:52.956933 18639 rpcz_store.cc:269] 0111 02:57:48.975234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.955542 (+3980308us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.955707 (+ 165us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250111 02:57:52.957960 20070 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
W20250111 02:57:52.954766 18638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 22) took 4143 ms (client timeout 598 ms). Trace:
W20250111 02:57:52.958242 18637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 38) took 3164 ms (client timeout 639 ms). Trace:
W20250111 02:57:52.941131 18633 rpcz_store.cc:269] 0111 02:57:45.634915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.940444 (+7305529us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.940581 (+ 137us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.958974 18637 rpcz_store.cc:269] 0111 02:57:49.794143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.957701 (+3163558us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.958125 (+ 424us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.959631 18633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 23) took 2864 ms (client timeout 607 ms). Trace:
W20250111 02:57:52.960186 18637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 39) took 2484 ms (client timeout 639 ms). Trace:
W20250111 02:57:52.960089 18633 rpcz_store.cc:269] 0111 02:57:50.095699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.959314 (+2863615us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.959548 (+ 234us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.960502 18637 rpcz_store.cc:269] 0111 02:57:50.476261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.960023 (+2483762us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.960136 (+ 113us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.958576 18638 rpcz_store.cc:269] 0111 02:57:48.811442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.954596 (+4143154us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.954716 (+ 120us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.958602 18640 rpcz_store.cc:269] 0111 02:57:37.079187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:37.079418 (+ 231us) service_pool.cc:225] Handling call
0111 02:57:52.953883 (+15874465us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:52.952332 18632 rpcz_store.cc:269] 0111 02:57:48.224158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.951663 (+4727505us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.951839 (+ 176us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250111 02:57:52.962951 18640 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [term 12 LEADER]: Rejecting Update request from peer c34b72b7c70343d3b6f3fae79ce4be6c for earlier term 11. Current term is 12. Ops: []
I20250111 02:57:52.963819 18640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55982 (request call id 35) took 4156 ms. Trace:
I20250111 02:57:52.962045 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 13 candidate_status { last_received { term: 11 index: 283 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:52.964108 18640 rpcz_store.cc:276] 0111 02:57:48.807005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.962381 (+4155376us) service_pool.cc:225] Handling call
0111 02:57:52.962717 (+ 336us) raft_consensus.cc:1466] Updating replica for 0 ops
0111 02:57:52.963533 (+ 816us) raft_consensus.cc:1692] Filling consensus response to leader.
0111 02:57:52.963705 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:52.954368 18635 rpcz_store.cc:269] 0111 02:57:48.191564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.953656 (+4762092us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.953814 (+ 158us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.964558 18638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55982 (request call id 40) took 937 ms (client timeout 978 ms). Trace:
W20250111 02:57:52.966198 18638 rpcz_store.cc:269] 0111 02:57:52.027563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.961733 (+934170us) service_pool.cc:225] Handling call
0111 02:57:52.964478 (+ 2745us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:57:52.967417 18693 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:52.968356 20682 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20250111 02:57:52.971129 18634 rpcz_store.cc:269] 0111 02:57:39.112066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:39.112708 (+ 642us) service_pool.cc:225] Handling call
0111 02:57:52.938548 (+13825840us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250111 02:57:52.948827 18631 rpcz_store.cc:269] 0111 02:57:48.179608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.948212 (+4768604us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.948364 (+ 152us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.933436 18636 rpcz_store.cc:269] 0111 02:57:43.587203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.932739 (+9345536us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.932923 (+ 184us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250111 02:57:52.961167 18633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55106 (request call id 24) took 2193 ms (client timeout 607 ms). Trace:
W20250111 02:57:52.975980 18633 rpcz_store.cc:269] 0111 02:57:50.768185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:52.960912 (+2192727us) service_pool.cc:212] Skipping call since client already timed out
0111 02:57:52.961098 (+ 186us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250111 02:57:53.141302 20654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 12 leader_uuid: "8784b9603f8544b79373158aace063af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } } }
I20250111 02:57:53.142217 20654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af [sys.catalog]: This master's current role is: LEADER
I20250111 02:57:53.231619 20070 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:57:53.403736 20070 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Loaded TSK: 0
I20250111 02:57:53.410710 20070 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:57:53.443018 20070 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:57:53.458671 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_26836b21f0f84973b442a4966f97d1ee [id=0037843fe90a422399b837b548ff45b9]
I20250111 02:57:53.459744 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609]
I20250111 02:57:53.460601 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_7653ac6289d7467590ac03b43df1b8bf [id=043fd6ad4f5c4a2685af5bbecda90328]
I20250111 02:57:53.461541 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_4b9e85e01e1843ce831205177cbb2e0f [id=0a118da8c5d0408980cbd0b33e0c2ed0]
I20250111 02:57:53.462445 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_28d700d6f8444cfeab3dc52dd2bec17f [id=0e5eefce9dfa453eb277f75189def89e]
I20250111 02:57:53.464223 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_bccba04b48cc408d845624ee7ed98e5f [id=1496848ec9594cad889312978436d9c4]
I20250111 02:57:53.465144 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_c398ec562f594aca8c1e62cde38c04f9 [id=14bc02cbe30c486782d8fa4396855a9a]
I20250111 02:57:53.466646 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_b6421db891c148c688d25610559fe16f [id=190cfd157c8d4bd19a8f40cfd80925c1]
I20250111 02:57:53.474969 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_dc22147bd8ba4ea18bc4f41b25b277c0 [id=1d14d647d8c24e31a84ce6ac7e2c217e]
I20250111 02:57:53.489750 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_37a329a4875c463ea093ad5ab23e9cf6 [id=23a0f0c1aaa149c68390a0d0327ff2c5]
I20250111 02:57:53.491822 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_e42719b273bf4b8f96fa48d1a3d6c8d8 [id=27f7d429d88b45b8b76d416b9dc582db]
I20250111 02:57:53.493985 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_c9582d98645645f3aa80a65f7652cba8 [id=2d5d8f8a7ade443f91a7da34add25a91]
I20250111 02:57:53.496465 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_a02982c32dcb409385f8f4e99b95423f [id=354a3a4b0b774fb9bc83623b2cb87f8a]
I20250111 02:57:53.498436 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]
I20250111 02:57:53.500362 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_baa2b0022e914aa5b485db90037be4af [id=3f31cb2959f340e3ae2656b63eaa52e6]
I20250111 02:57:53.501641 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_4c9aebb06d554107a5448d968fdb0436 [id=3fab00618e6548de85b5d20adee4e9ac]
I20250111 02:57:53.503764 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_1ab2b1f61ced4c97ad5e44f18b6e8699 [id=40eb021e024747f0a173bf87538b7281]
I20250111 02:57:53.504999 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_bdacd508498241408a549fdbc6704512 [id=46a029216fdb4496bf9df1d776ccdf7e]
I20250111 02:57:53.506942 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_a68194fdb909449f878695d09b81c753 [id=49aa9df32f8e4001b66d751832a64534]
I20250111 02:57:53.508153 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_bf50e4c0392544f8a24b861d3edaf46e [id=4b08fa373b0b460094c09b4d3353951c]
I20250111 02:57:53.509325 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_4f96c1244592444a994b5c5631eaaadc [id=4c817a432cf649799a2589f12a320ebd]
I20250111 02:57:53.510557 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_39f0114c91ef4d1c86ef0cf522e31e11 [id=5189b95a9b0e4b47ac0d7768e08e902b]
I20250111 02:57:53.511736 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_dacea9000ff840659052c41c94a8bfaf [id=51bd1567dc9c43b2877791b72c2c2046]
I20250111 02:57:53.513582 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_535081a0b0ff46998f523371377b75f9 [id=531ca04d38e9427eb955326135436b79]
I20250111 02:57:53.514775 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_97e52a18fbd44f189d46ba9b26cd9033 [id=566e89e7552f4ce08fba16fe299b9ccf]
I20250111 02:57:53.516686 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_0dcad07d56d843e59f351d933e625a21 [id=57e820dd99764a578d0c50f5f250aad4]
I20250111 02:57:53.520424 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]
I20250111 02:57:53.522486 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_85a0376bcaf24d0081e780e8c19764b8 [id=5f97e9e0988848319a9c8aaca6aad982]
I20250111 02:57:53.523891 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_c2fa721b96ba4e9ca293ff4b5c455447 [id=64d78652fc6f44778133bd3157964db5]
I20250111 02:57:53.525460 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_f36240205653470e9a67400b787ce860 [id=69f6e7b3ec40423a9d0bef5487ffe84a]
I20250111 02:57:53.527621 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_52d947397a774f96a4fdb5087da4e19a [id=7196ac9c0c4d4dada6c94803e1220caa]
I20250111 02:57:53.528999 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_4e3a79a5d2184687a8c51fd2702c5fd0 [id=73c59c0d9864497fbc9f19e265d8283f]
I20250111 02:57:53.530964 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_b9769d513090402bb596c949defad0df [id=76e72ea89f6e408fbaeda0efc7e79770]
I20250111 02:57:53.532966 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_6a0ed66485f640e9a2c11b62b5cbd239 [id=7b936eef4300409e908cc92b1453b500]
I20250111 02:57:53.536978 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_8eae98e1e49c4f6c9e52644256812eaa [id=7dfae414add74b6491f2f7595a551342]
I20250111 02:57:53.543784 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204]
I20250111 02:57:53.545152 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_53d44be8eaeb4e59a8e3454aac56daca [id=8de61815471842e593a7d29efffc443c]
I20250111 02:57:53.546010 20689 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:53.546417 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0]
I20250111 02:57:53.547712 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_5e7c139097914dd7b7c8e55058d5290d [id=9143d1d3b8754ac78d615a1cc4601d86]
I20250111 02:57:53.547034 20689 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:53.549616 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605]
I20250111 02:57:53.550801 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_c0ddb0fbf2f347c0b82acf9060722ed2 [id=964e4311f5ed4578a6f7285a9d2bbafa]
I20250111 02:57:53.552150 20689 leader_election.cc:290] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 8784b9603f8544b79373158aace063af (127.18.20.125:40359), c34b72b7c70343d3b6f3fae79ce4be6c (127.18.20.124:34593)
I20250111 02:57:53.552507 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_02b5991089a34b0b8ee6039ccc092c93 [id=977f59b134154bc396bf42bc1e44e717]
I20250111 02:57:53.555768 18717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 13 candidate_status { last_received { term: 12 index: 283 } } ignore_live_leader: false dest_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" is_pre_election: true
I20250111 02:57:53.575743 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_98958a28faeb43e8a3b34fcf20d936aa [id=98e24956666a413cbf4564438a594ef0]
I20250111 02:57:53.579154 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_f9d19027fe814825a6c3906baef77ff3 [id=9a4fbc1c2c2c44c182f9935156885c5a]
I20250111 02:57:53.587481 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_857f04d0de7e47a4828db9a8997f1009 [id=9cf908e0c7d44b60b88a32cf2d6368a0]
I20250111 02:57:53.589978 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7]
I20250111 02:57:53.591470 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_1431a031eea3496b9764a373a012a367 [id=a0cdf8fe647d4c84a9ce97df9ce9a0ce]
I20250111 02:57:53.593415 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_fc5930652905494584fde76fff8629d8 [id=a25d2996d4d64d91980bb5a067f5b118]
I20250111 02:57:53.594650 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_bdf982e1d6904892ac08ff16ed20a636 [id=a2b6467b58754b85ab0481a9e8aa0ace]
I20250111 02:57:53.595930 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19]
I20250111 02:57:53.597856 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_ee18021e6a544ed6b86b624aec8c39be [id=a9538f81833c4c0d8d02b6271a67f711]
I20250111 02:57:53.599407 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_2c181f66f28741b7b388a7b563b7d2f5 [id=ac0e88a97cf740fd8df3e6cbd81888b5]
I20250111 02:57:53.600653 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3]
I20250111 02:57:53.603232 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_6ad0b827920c42c68113310190019df1 [id=b25314ef99fc4e0ba80c6391223d2a3d]
I20250111 02:57:53.604461 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_b19a186d285d4b5b845727344f7d155b [id=b28b429fe5844cd6ac6f4de349c9449b]
I20250111 02:57:53.605664 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0]
I20250111 02:57:53.606813 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_dfe42fac0a764052a8d2f6add8882b64 [id=b42908f2a2c14d0a84027cb7a1d775b7]
I20250111 02:57:53.610649 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe]
I20250111 02:57:53.612541 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_24e5a7f819cc45dbb7efbd7286ad7005 [id=c04c27fb52054de185e1f3075f89a59a]
I20250111 02:57:53.614686 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_23a3f1c40c15458e93c2ef79a3c9e9e2 [id=c12add3740174f66996466994ebe9971]
I20250111 02:57:53.616003 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_bf5cf724093d4691a60379b87cfab2f0 [id=c151a212f267426a9cb6180c938a5815]
I20250111 02:57:53.617228 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_aea3cbca3247424c9a12e310135fe0b9 [id=c151ba8ccc454372b4ad7621b6bc7aeb]
I20250111 02:57:53.618503 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927]
I20250111 02:57:53.619625 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_cce40438ec8e4527b01088af6b7b0981 [id=cb42d3930a9a44bfb5dc40feb979d505]
I20250111 02:57:53.621611 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5]
I20250111 02:57:53.622828 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_d7e00ace61e545129a2e99093dba16db [id=d0e7184505c24fc5b9960e8b6d1c955d]
I20250111 02:57:53.624081 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3]
I20250111 02:57:53.625941 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_6823aef149894f49822fe303c036446d [id=d5ffe760744748a98f80f25b8092f9fa]
I20250111 02:57:53.627148 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_5e21879e67544c71af627f795fd6d940 [id=d6c0153a97cd42d49f534f48fc5b8287]
I20250111 02:57:53.629119 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_813d4e8bb0f34a46bf96ed3bee6ca86f [id=d790288a097a4416b68b93db6ac847fd]
I20250111 02:57:53.631033 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_23d632d4476c46b78d27619201589641 [id=db5ffbbe4fe7424f943fbaca17103d84]
I20250111 02:57:53.632299 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_ea068446b5434b3ca5f0afa8386c0655 [id=dc09e842add34ef099edfd1296255c4b]
I20250111 02:57:53.634100 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_a7b2b5af5dfb439a84159b75de0d03b1 [id=dc7854580f744d4083f6aa3f0f0f7bf2]
I20250111 02:57:53.636270 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_f9181ca352384593b93119fb3e0cd812 [id=e0ac9d601c464a3daaaa63c4cb90e471]
I20250111 02:57:53.637459 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_c4faffc9659848ea9bff581e8d88d40e [id=e709d8f708144de8a18d241f0e0306a0]
I20250111 02:57:53.641353 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_799d0b8328de40a7bacb58ea433fec72 [id=e9efdc6c94574ddca092dd6b9f3b8419]
I20250111 02:57:53.642612 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_902d917b19634ad5b20fadd455c0ba88 [id=ec311d46e9224fd38d1864b43717da4a]
I20250111 02:57:53.644444 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_b44f1ebbaf7045e2bcfefcee1c91c0d9 [id=ec5cf5f9b6234008914c0ed664260dcb]
I20250111 02:57:53.646358 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_e091dc5f47d9492aa86ff883e04fe415 [id=ee752760552944289c461f1c4c50d90f]
I20250111 02:57:53.647532 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_49e4dd58d29f4cb6a0ed4f85138713b8 [id=f1373922978a4598b888e1863525942e]
I20250111 02:57:53.650560 20070 catalog_manager.cc:670] Loaded metadata for table default.Table_7ee7d20cfb7e4985bdd8539d15e354fe [id=ff66cd83751a45a09d8012cc8a9f1758]
I20250111 02:57:53.657286 20690 raft_consensus.cc:491] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 8784b9603f8544b79373158aace063af)
I20250111 02:57:53.657954 20690 raft_consensus.cc:513] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 45115 } } peers { permanent_uuid: "8784b9603f8544b79373158aace063af" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 40359 } } peers { permanent_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 34593 } }
I20250111 02:57:53.661175 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 13 candidate_status { last_received { term: 12 index: 283 } } ignore_live_leader: false dest_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" is_pre_election: true
I20250111 02:57:53.665339 20690 leader_election.cc:290] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e336d4ddf044bf5b0b6e4f5af634727 (127.18.20.126:45115), 8784b9603f8544b79373158aace063af (127.18.20.125:40359)
I20250111 02:57:53.673774 20070 tablet_loader.cc:96] loaded metadata for tablet 0158a8849bf44a4fbb5daca22883478f (table default.Table_c2fa721b96ba4e9ca293ff4b5c455447 [id=64d78652fc6f44778133bd3157964db5])
I20250111 02:57:53.674970 20070 tablet_loader.cc:96] loaded metadata for tablet 0255034337f24337be838425136d8b4c (table default.Table_813d4e8bb0f34a46bf96ed3bee6ca86f [id=d790288a097a4416b68b93db6ac847fd])
I20250111 02:57:53.679225 20070 tablet_loader.cc:96] loaded metadata for tablet 07f15096ea484b499fbdecec74eb6f66 (table default.Table_7653ac6289d7467590ac03b43df1b8bf [id=043fd6ad4f5c4a2685af5bbecda90328])
I20250111 02:57:53.680409 20070 tablet_loader.cc:96] loaded metadata for tablet 0b6687f39d4246b49c0c3eb904b11c1a (table default.Table_857f04d0de7e47a4828db9a8997f1009 [id=9cf908e0c7d44b60b88a32cf2d6368a0])
I20250111 02:57:53.681613 20070 tablet_loader.cc:96] loaded metadata for tablet 0c669a3ff8aa46fa8b04b86ed3bfde49 (table default.Table_fc5930652905494584fde76fff8629d8 [id=a25d2996d4d64d91980bb5a067f5b118])
I20250111 02:57:53.682646 20070 tablet_loader.cc:96] loaded metadata for tablet 0d4ba33836f04ed19d67b8bc161e57c4 (table default.Table_bf5cf724093d4691a60379b87cfab2f0 [id=c151a212f267426a9cb6180c938a5815])
I20250111 02:57:53.685920 20070 tablet_loader.cc:96] loaded metadata for tablet 12065effb54b4d4a94349641109b578c (table default.Table_d7e00ace61e545129a2e99093dba16db [id=d0e7184505c24fc5b9960e8b6d1c955d])
I20250111 02:57:53.688375 20070 tablet_loader.cc:96] loaded metadata for tablet 16d88c5da4f345c3a43a1ef00f82856b (table default.Table_a7b2b5af5dfb439a84159b75de0d03b1 [id=dc7854580f744d4083f6aa3f0f0f7bf2])
I20250111 02:57:53.691313 20070 tablet_loader.cc:96] loaded metadata for tablet 1b69a8c2caea4bddac3f539733bcd3af (table default.Table_bf50e4c0392544f8a24b861d3edaf46e [id=4b08fa373b0b460094c09b4d3353951c])
I20250111 02:57:53.693197 20070 tablet_loader.cc:96] loaded metadata for tablet 1ef92ac716824107916a1ea4fd49d5aa (table default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609])
I20250111 02:57:53.694394 20070 tablet_loader.cc:96] loaded metadata for tablet 2009d6f54e2842e8be3af2cc9147dc2c (table default.Table_24e5a7f819cc45dbb7efbd7286ad7005 [id=c04c27fb52054de185e1f3075f89a59a])
I20250111 02:57:53.695514 20070 tablet_loader.cc:96] loaded metadata for tablet 218b87091d9546ba859f954a8f9e089e (table default.Table_c398ec562f594aca8c1e62cde38c04f9 [id=14bc02cbe30c486782d8fa4396855a9a])
I20250111 02:57:53.696651 20070 tablet_loader.cc:96] loaded metadata for tablet 220da826a0f140d59193e8ad72096b60 (table default.Table_5e7c139097914dd7b7c8e55058d5290d [id=9143d1d3b8754ac78d615a1cc4601d86])
I20250111 02:57:53.698697 20070 tablet_loader.cc:96] loaded metadata for tablet 237ea33846044127987c1d81a3203553 (table default.Table_b9769d513090402bb596c949defad0df [id=76e72ea89f6e408fbaeda0efc7e79770])
I20250111 02:57:53.701298 20070 tablet_loader.cc:96] loaded metadata for tablet 29f27ed2c53d43d7a51ee2e45afaf9fc (table default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7])
I20250111 02:57:53.705974 20070 tablet_loader.cc:96] loaded metadata for tablet 2f11541b436e42b7b0728da2aaec48f7 (table default.Table_f36240205653470e9a67400b787ce860 [id=69f6e7b3ec40423a9d0bef5487ffe84a])
I20250111 02:57:53.707841 20070 tablet_loader.cc:96] loaded metadata for tablet 315ac86194b04802909a573876fce64b (table default.Table_ea068446b5434b3ca5f0afa8386c0655 [id=dc09e842add34ef099edfd1296255c4b])
I20250111 02:57:53.710126 20070 tablet_loader.cc:96] loaded metadata for tablet 350f7c7f14004bb4bc3cb2338c7f172f (table default.Table_28d700d6f8444cfeab3dc52dd2bec17f [id=0e5eefce9dfa453eb277f75189def89e])
I20250111 02:57:53.711558 20070 tablet_loader.cc:96] loaded metadata for tablet 3514d795a0f24adcaa2accc0cf05b142 (table default.Table_4c9aebb06d554107a5448d968fdb0436 [id=3fab00618e6548de85b5d20adee4e9ac])
I20250111 02:57:53.712790 20070 tablet_loader.cc:96] loaded metadata for tablet 36f71f8dd2894d40b3bfdd20c1453787 (table default.Table_cce40438ec8e4527b01088af6b7b0981 [id=cb42d3930a9a44bfb5dc40feb979d505])
I20250111 02:57:53.716037 20070 tablet_loader.cc:96] loaded metadata for tablet 3c57d85390324bbc9ea30a709faec01c (table default.Table_52d947397a774f96a4fdb5087da4e19a [id=7196ac9c0c4d4dada6c94803e1220caa])
I20250111 02:57:53.717105 20070 tablet_loader.cc:96] loaded metadata for tablet 3d873a0faf4a4645a3969067ae0583dc (table default.Table_c9582d98645645f3aa80a65f7652cba8 [id=2d5d8f8a7ade443f91a7da34add25a91])
I20250111 02:57:53.719079 20070 tablet_loader.cc:96] loaded metadata for tablet 40fdec982cf643c68ef90798f4f65ad4 (table default.Table_94ff268a8e3942ae997a5d44c55b4f43 [id=c594b9f9a14b4f97949baf078e58b927])
I20250111 02:57:53.722545 20070 tablet_loader.cc:96] loaded metadata for tablet 49616e4e7cc647509e72cde57b87d6fa (table default.Table_c4faffc9659848ea9bff581e8d88d40e [id=e709d8f708144de8a18d241f0e0306a0])
I20250111 02:57:53.725968 20070 tablet_loader.cc:96] loaded metadata for tablet 56fb0bf949904208af003f08caae7594 (table default.Table_23a3f1c40c15458e93c2ef79a3c9e9e2 [id=c12add3740174f66996466994ebe9971])
I20250111 02:57:53.727268 20070 tablet_loader.cc:96] loaded metadata for tablet 5a4bc8e9c0c947f6861a01d3e3bd7f31 (table default.Table_a68194fdb909449f878695d09b81c753 [id=49aa9df32f8e4001b66d751832a64534])
I20250111 02:57:53.732715 20070 tablet_loader.cc:96] loaded metadata for tablet 63aef98c83d94918bb3163d491870084 (table default.Table_98958a28faeb43e8a3b34fcf20d936aa [id=98e24956666a413cbf4564438a594ef0])
I20250111 02:57:53.733983 20070 tablet_loader.cc:96] loaded metadata for tablet 64abb30da7fc452cad774bbe59b498f4 (table default.Table_26836b21f0f84973b442a4966f97d1ee [id=0037843fe90a422399b837b548ff45b9])
I20250111 02:57:53.735212 20070 tablet_loader.cc:96] loaded metadata for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038])
I20250111 02:57:53.737237 20070 tablet_loader.cc:96] loaded metadata for tablet 6ff06134303d4cb0b4292fe1c0c2d443 (table default.Table_2c181f66f28741b7b388a7b563b7d2f5 [id=ac0e88a97cf740fd8df3e6cbd81888b5])
I20250111 02:57:53.738425 20070 tablet_loader.cc:96] loaded metadata for tablet 7145efe5c0094b70be2cd6481b770248 (table default.Table_7ee7d20cfb7e4985bdd8539d15e354fe [id=ff66cd83751a45a09d8012cc8a9f1758])
I20250111 02:57:53.741860 20070 tablet_loader.cc:96] loaded metadata for tablet 788c34eff1884d979add7e09615c02ae (table default.Table_dfe42fac0a764052a8d2f6add8882b64 [id=b42908f2a2c14d0a84027cb7a1d775b7])
I20250111 02:57:53.743916 20070 tablet_loader.cc:96] loaded metadata for tablet 79f4500d3beb4c9bb20f84c661c3758b (table default.Table_bdacd508498241408a549fdbc6704512 [id=46a029216fdb4496bf9df1d776ccdf7e])
I20250111 02:57:53.744975 20070 tablet_loader.cc:96] loaded metadata for tablet 7c3c5b038d994d07abfea701194e8ad9 (table default.Table_bdf982e1d6904892ac08ff16ed20a636 [id=a2b6467b58754b85ab0481a9e8aa0ace])
I20250111 02:57:53.746202 20070 tablet_loader.cc:96] loaded metadata for tablet 7ea78feee64340e39404cba0f1bf04db (table default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5])
I20250111 02:57:53.747653 20070 tablet_loader.cc:96] loaded metadata for tablet 8414653def014ce98bc38d87aac75bae (table default.Table_6254852e82bc4e078fac113005b83d10 [id=8b793882ed0c4f68baca0241b6cd0204])
I20250111 02:57:53.748837 20070 tablet_loader.cc:96] loaded metadata for tablet 845da3ee0863430ea3fe9de8c84f2c17 (table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3])
I20250111 02:57:53.750069 20070 tablet_loader.cc:96] loaded metadata for tablet 855dac47872d4c5fbf8f5e38b588c7fe (table default.Table_49e4dd58d29f4cb6a0ed4f85138713b8 [id=f1373922978a4598b888e1863525942e])
I20250111 02:57:53.752720 20070 tablet_loader.cc:96] loaded metadata for tablet 8d20e8099b964fbc9e8e41447772d8d5 (table default.Table_a02982c32dcb409385f8f4e99b95423f [id=354a3a4b0b774fb9bc83623b2cb87f8a])
I20250111 02:57:53.754073 20070 tablet_loader.cc:96] loaded metadata for tablet 8eb87813c28742edad99470c84269ef4 (table default.Table_f9181ca352384593b93119fb3e0cd812 [id=e0ac9d601c464a3daaaa63c4cb90e471])
I20250111 02:57:53.755220 20070 tablet_loader.cc:96] loaded metadata for tablet 900d4d665c4a4048bc6989ba63391cc1 (table default.Table_baa2b0022e914aa5b485db90037be4af [id=3f31cb2959f340e3ae2656b63eaa52e6])
I20250111 02:57:53.758134 20070 tablet_loader.cc:96] loaded metadata for tablet 9192160e931445f1acd207604192f759 (table default.Table_bccba04b48cc408d845624ee7ed98e5f [id=1496848ec9594cad889312978436d9c4])
I20250111 02:57:53.760062 20070 tablet_loader.cc:96] loaded metadata for tablet 933828fd6fb041e9a4889f4ea8a3c5ff (table default.Table_dc22147bd8ba4ea18bc4f41b25b277c0 [id=1d14d647d8c24e31a84ce6ac7e2c217e])
I20250111 02:57:53.761277 20070 tablet_loader.cc:96] loaded metadata for tablet 934c900baf1b4000af7d7ec60cbb3591 (table default.Table_4f96c1244592444a994b5c5631eaaadc [id=4c817a432cf649799a2589f12a320ebd])
I20250111 02:57:53.763095 20070 tablet_loader.cc:96] loaded metadata for tablet 94bb9089c71b49bea9c0f4d94fc22fb1 (table default.Table_b19a186d285d4b5b845727344f7d155b [id=b28b429fe5844cd6ac6f4de349c9449b])
I20250111 02:57:53.765046 20070 tablet_loader.cc:96] loaded metadata for tablet 9d6de202f4e547a0b8b90cfa09ad319a (table default.Table_6823aef149894f49822fe303c036446d [id=d5ffe760744748a98f80f25b8092f9fa])
I20250111 02:57:53.766130 20070 tablet_loader.cc:96] loaded metadata for tablet 9ee884f4d31f46c197c99bf72f1f19bf (table default.Table_85a0376bcaf24d0081e780e8c19764b8 [id=5f97e9e0988848319a9c8aaca6aad982])
I20250111 02:57:53.768734 20070 tablet_loader.cc:96] loaded metadata for tablet a2eae136623e4802b083d6526c6867c9 (table default.Table_97e52a18fbd44f189d46ba9b26cd9033 [id=566e89e7552f4ce08fba16fe299b9ccf])
I20250111 02:57:53.779541 20070 tablet_loader.cc:96] loaded metadata for tablet a6e4b222bf974a8d80b25141a5686761 (table default.Table_5e21879e67544c71af627f795fd6d940 [id=d6c0153a97cd42d49f534f48fc5b8287])
I20250111 02:57:53.780843 20070 tablet_loader.cc:96] loaded metadata for tablet a983a670c2104398ac1a159ac3a1da7c (table default.Table_e1f7351012894b2e9e3dde1c1006874d [id=a37b721314134c86b39d1f80b7359f19])
I20250111 02:57:53.782009 20070 tablet_loader.cc:96] loaded metadata for tablet aaab428f798a44f69c80e5792d56b920 (table default.Table_e42719b273bf4b8f96fa48d1a3d6c8d8 [id=27f7d429d88b45b8b76d416b9dc582db])
I20250111 02:57:53.783138 20070 tablet_loader.cc:96] loaded metadata for tablet aca35147034446b3882b5244923bddc7 (table default.Table_1431a031eea3496b9764a373a012a367 [id=a0cdf8fe647d4c84a9ce97df9ce9a0ce])
I20250111 02:57:53.784260 20070 tablet_loader.cc:96] loaded metadata for tablet adf61f6cb63d4e8ea46b440e7ecce00d (table default.Table_c0ddb0fbf2f347c0b82acf9060722ed2 [id=964e4311f5ed4578a6f7285a9d2bbafa])
I20250111 02:57:53.785984 20070 tablet_loader.cc:96] loaded metadata for tablet aec6dffdaa6b4798b7827a6a445601ca (table default.Table_0dcad07d56d843e59f351d933e625a21 [id=57e820dd99764a578d0c50f5f250aad4])
I20250111 02:57:53.787891 20070 tablet_loader.cc:96] loaded metadata for tablet b2f561cfd4f34426b7e42027e53b3ff1 (table default.Table_37a329a4875c463ea093ad5ab23e9cf6 [id=23a0f0c1aaa149c68390a0d0327ff2c5])
I20250111 02:57:53.789001 20070 tablet_loader.cc:96] loaded metadata for tablet b4273afc264843029ed0baf1fc5c8a3e (table default.Table_4e3a79a5d2184687a8c51fd2702c5fd0 [id=73c59c0d9864497fbc9f19e265d8283f])
I20250111 02:57:53.790151 20070 tablet_loader.cc:96] loaded metadata for tablet b5148fa2e6504561bd1b1a19e48993d7 (table default.Table_c8ee4857085f45dfb5adc86d85bead9d [id=8eeffc36d38543339b1c18fa18b0bac0])
I20250111 02:57:53.794113 20070 tablet_loader.cc:96] loaded metadata for tablet bc0bd71c6f4d4c90a4f3ea747af02165 (table default.Table_535081a0b0ff46998f523371377b75f9 [id=531ca04d38e9427eb955326135436b79])
I20250111 02:57:53.797768 20070 tablet_loader.cc:96] loaded metadata for tablet c8f0c42e06d4428284fa006fe0062cf0 (table default.Table_6ad0b827920c42c68113310190019df1 [id=b25314ef99fc4e0ba80c6391223d2a3d])
I20250111 02:57:53.799000 20070 tablet_loader.cc:96] loaded metadata for tablet cb96635f30b34de48870daf5166813bc (table default.Table_39f0114c91ef4d1c86ef0cf522e31e11 [id=5189b95a9b0e4b47ac0d7768e08e902b])
I20250111 02:57:53.800280 20070 tablet_loader.cc:96] loaded metadata for tablet cc7158a7fc624fa4bc816ac3721c610d (table default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605])
I20250111 02:57:53.801277 20070 tablet_loader.cc:96] loaded metadata for tablet d2fc07e0474e4fc5a46f2350229ca31f (table default.Table_6a0ed66485f640e9a2c11b62b5cbd239 [id=7b936eef4300409e908cc92b1453b500])
I20250111 02:57:53.802424 20070 tablet_loader.cc:96] loaded metadata for tablet d3b7616fe56f4e5fb49bdde6961fa008 (table default.Table_1ab2b1f61ced4c97ad5e44f18b6e8699 [id=40eb021e024747f0a173bf87538b7281])
I20250111 02:57:53.803601 20070 tablet_loader.cc:96] loaded metadata for tablet d6da575f052d4b20a8e41f0bd9dab7e4 (table default.Table_02b5991089a34b0b8ee6039ccc092c93 [id=977f59b134154bc396bf42bc1e44e717])
I20250111 02:57:53.805511 20070 tablet_loader.cc:96] loaded metadata for tablet d837bd001aa649e9b3ca94f7cdd9f0fe (table default.Table_b6421db891c148c688d25610559fe16f [id=190cfd157c8d4bd19a8f40cfd80925c1])
I20250111 02:57:53.806645 20070 tablet_loader.cc:96] loaded metadata for tablet d83cfd85a52f438dabe49b6223d5090f (table default.Table_799d0b8328de40a7bacb58ea433fec72 [id=e9efdc6c94574ddca092dd6b9f3b8419])
I20250111 02:57:53.807852 20070 tablet_loader.cc:96] loaded metadata for tablet d8ef3f45e5784a29b3b32cc0de6bac84 (table default.Table_8776da962d214139886beb69df33db00 [id=b9b226a7c0da4da3aca14ddcba6ab5fe])
I20250111 02:57:53.808892 20070 tablet_loader.cc:96] loaded metadata for tablet dea4db154779483cb069118e3f3789b0 (table default.Table_8eae98e1e49c4f6c9e52644256812eaa [id=7dfae414add74b6491f2f7595a551342])
I20250111 02:57:53.810046 20070 tablet_loader.cc:96] loaded metadata for tablet df1870cceafe463b8d024e8b065850e0 (table default.Table_aea3cbca3247424c9a12e310135fe0b9 [id=c151ba8ccc454372b4ad7621b6bc7aeb])
I20250111 02:57:53.811232 20070 tablet_loader.cc:96] loaded metadata for tablet e01771c56ea5498a876d3bb865c752a2 (table default.Table_dacea9000ff840659052c41c94a8bfaf [id=51bd1567dc9c43b2877791b72c2c2046])
I20250111 02:57:53.812429 20070 tablet_loader.cc:96] loaded metadata for tablet e46894710a72407993b36801f3b90eb0 (table default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3])
I20250111 02:57:53.814505 20070 tablet_loader.cc:96] loaded metadata for tablet e4bd42ad66064bcaa043746081f1c759 (table default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7])
I20250111 02:57:53.815727 20070 tablet_loader.cc:96] loaded metadata for tablet ea76b354a75041e888cf862250e4947b (table default.Table_b44f1ebbaf7045e2bcfefcee1c91c0d9 [id=ec5cf5f9b6234008914c0ed664260dcb])
I20250111 02:57:53.818980 20070 tablet_loader.cc:96] loaded metadata for tablet f1802e7349934fbb9d5f7eda37c3df20 (table default.Table_23038750b6334f158755d8f4c06190f8 [id=b2f59358b7c34e0b88ae3d165f2879b0])
I20250111 02:57:53.820201 20070 tablet_loader.cc:96] loaded metadata for tablet f31ad2e1895d4a26b8e9f7cd7366ef81 (table default.Table_23d632d4476c46b78d27619201589641 [id=db5ffbbe4fe7424f943fbaca17103d84])
I20250111 02:57:53.822707 20070 tablet_loader.cc:96] loaded metadata for tablet f778763e59a94969b3f62be87e2fd867 (table default.Table_902d917b19634ad5b20fadd455c0ba88 [id=ec311d46e9224fd38d1864b43717da4a])
I20250111 02:57:53.823755 20070 tablet_loader.cc:96] loaded metadata for tablet f9092a3453f94abbbf4bdd66a57d42cc (table default.Table_ee18021e6a544ed6b86b624aec8c39be [id=a9538f81833c4c0d8d02b6271a67f711])
I20250111 02:57:53.824738 20070 tablet_loader.cc:96] loaded metadata for tablet f9ec6b2a27744ee1aef0efdc98d8757c (table default.Table_f9d19027fe814825a6c3906baef77ff3 [id=9a4fbc1c2c2c44c182f9935156885c5a])
I20250111 02:57:53.825866 20070 tablet_loader.cc:96] loaded metadata for tablet fa86b1e6eca7403c9a9c52592b8fb636 (table default.Table_4b9e85e01e1843ce831205177cbb2e0f [id=0a118da8c5d0408980cbd0b33e0c2ed0])
I20250111 02:57:53.826898 20070 tablet_loader.cc:96] loaded metadata for tablet fb9b1c7e252d40e29e6d16311d145968 (table default.Table_e091dc5f47d9492aa86ff883e04fe415 [id=ee752760552944289c461f1c4c50d90f])
I20250111 02:57:53.828023 20070 tablet_loader.cc:96] loaded metadata for tablet ffd063f8889b4bcdb9656466f0b144f3 (table default.Table_53d44be8eaeb4e59a8e3454aac56daca [id=8de61815471842e593a7d29efffc443c])
I20250111 02:57:53.841528 20070 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:57:53.859582 20070 catalog_manager.cc:1260] Loaded cluster ID: 505f17df54394056975cd03f4ae103e8
I20250111 02:57:53.859922 20070 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:57:53.879280 20070 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:57:53.893419 18633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e336d4ddf044bf5b0b6e4f5af634727" candidate_term: 13 candidate_status { last_received { term: 12 index: 283 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:53.895188 19594 leader_election.cc:304] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e336d4ddf044bf5b0b6e4f5af634727; no voters: 8784b9603f8544b79373158aace063af, c34b72b7c70343d3b6f3fae79ce4be6c
I20250111 02:57:53.896445 20689 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 1e336d4ddf044bf5b0b6e4f5af634727 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250111 02:57:53.896729 20070 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Loaded TSK: 0
I20250111 02:57:53.897647 20070 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:57:53.897715 18636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34b72b7c70343d3b6f3fae79ce4be6c" candidate_term: 13 candidate_status { last_received { term: 12 index: 283 } } ignore_live_leader: false dest_uuid: "8784b9603f8544b79373158aace063af" is_pre_election: true
I20250111 02:57:53.899888 18693 leader_election.cc:304] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c34b72b7c70343d3b6f3fae79ce4be6c; no voters: 1e336d4ddf044bf5b0b6e4f5af634727, 8784b9603f8544b79373158aace063af
I20250111 02:57:53.901363 20690 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c34b72b7c70343d3b6f3fae79ce4be6c [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250111 02:57:53.963456 18626 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_e2a51ae779294deb90786269e7a8f13d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:57:53.964579 18630 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_a02982c32dcb409385f8f4e99b95423f" } new_table_name: "default.Table_6cdffa2e029846c3b95c8440c221426c" modify_external_catalogs: true
W20250111 02:57:53.966449 18626 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2a51ae779294deb90786269e7a8f13d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:53.973287 18622 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_5b39045192764dd8a67ddaab57d02a5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:53.974918 18622 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b39045192764dd8a67ddaab57d02a5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:53.982133 18625 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_ccb78dfa1e94441ca883240c2d85ca5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:53.983819 18625 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccb78dfa1e94441ca883240c2d85ca5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:57:53.984853 18986 consensus_peers.cc:487] T 2cd364f9a6894d19826deb59a8d8a612 P 54628fbcec2f4359aa9e26a734d4d29f -> Peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Couldn't send request to peer d0cedab82b154d62b78809edc6f28bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cd364f9a6894d19826deb59a8d8a612. This is attempt 11: this message will repeat every 5th retry.
I20250111 02:57:53.988135 19100 heartbeater.cc:502] Master 127.18.20.125:40359 was elected leader, sending a full tablet report...
I20250111 02:57:54.001349 18624 master_service.cc:946] ReplaceTablet: received request to replace tablet d6da575f052d4b20a8e41f0bd9dab7e4 from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:54.007272 18621 master_service.cc:946] ReplaceTablet: received request to replace tablet 1ef92ac716824107916a1ea4fd49d5aa from {username='slave'} at 127.0.0.1:47522
I20250111 02:57:54.015045 18623 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47522:
name: "default.Table_5d8fd938fc9749e8a833fe51b755d938"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:57:54.017450 18623 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d8fd938fc9749e8a833fe51b755d938 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:57:54.026262 18629 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_e1f7351012894b2e9e3dde1c1006874d" } new_table_name: "default.Table_745da7eb612249d8b494bee5c97df63a" modify_external_catalogs: true
I20250111 02:57:54.032131 18627 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_799d0b8328de40a7bacb58ea433fec72" } modify_external_catalogs: true
I20250111 02:57:54.032859 18627 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_799d0b8328de40a7bacb58ea433fec72" } modify_external_catalogs: true
I20250111 02:57:54.064985 18628 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_7653ac6289d7467590ac03b43df1b8bf" } modify_external_catalogs: true
I20250111 02:57:54.065630 18628 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47522:
table { table_name: "default.Table_7653ac6289d7467590ac03b43df1b8bf" } modify_external_catalogs: true
W20250111 02:57:54.190024 18630 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d20e8099b964fbc9e8e41447772d8d5 has no consensus state
W20250111 02:57:54.212049 18617 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 8d20e8099b964fbc9e8e41447772d8d5 (table default.Table_6cdffa2e029846c3b95c8440c221426c [id=354a3a4b0b774fb9bc83623b2cb87f8a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8d20e8099b964fbc9e8e41447772d8d5 has no consensus state
I20250111 02:57:54.306143 20694 raft_consensus.cc:491] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:54.306731 20694 raft_consensus.cc:513] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } }
W20250111 02:57:54.311550 18841 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): Not found: Tablet not found: 32afc2f158254811836fcf00f7f409ed
I20250111 02:57:54.312145 20694 leader_election.cc:290] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
W20250111 02:57:54.314441 18842 leader_election.cc:343] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): Not found: Tablet not found: 32afc2f158254811836fcf00f7f409ed
I20250111 02:57:54.314870 18842 leader_election.cc:304] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221604733fe4fc0a1fc32de8d6af71c; no voters: 54628fbcec2f4359aa9e26a734d4d29f, d0cedab82b154d62b78809edc6f28bfa
I20250111 02:57:54.315647 20694 raft_consensus.cc:2743] T 32afc2f158254811836fcf00f7f409ed P 8221604733fe4fc0a1fc32de8d6af71c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:57:54.361667 18513 master-stress-test.cc:579] Killed test thread 1/10
I20250111 02:57:54.362107 18513 master-stress-test.cc:577] Killing test thread 2/10
I20250111 02:57:54.368095 18621 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 1ef92ac716824107916a1ea4fd49d5aa
I20250111 02:57:54.369725 18621 catalog_manager.cc:6596] ReplaceTablet: tablet 1ef92ac716824107916a1ea4fd49d5aa deleted and replaced by tablet 5766b6e8bc2d4d9683abd39fba8d2089
I20250111 02:57:54.370690 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef92ac716824107916a1ea4fd49d5aa with delete_type TABLET_DATA_DELETED (replaced by tablet 5766b6e8bc2d4d9683abd39fba8d2089) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:54.374074 19034 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef92ac716824107916a1ea4fd49d5aa with delete_type TABLET_DATA_DELETED (replaced by tablet 5766b6e8bc2d4d9683abd39fba8d2089) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:54.381575 19173 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef92ac716824107916a1ea4fd49d5aa with delete_type TABLET_DATA_DELETED (replaced by tablet 5766b6e8bc2d4d9683abd39fba8d2089) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:54.420310 18513 master-stress-test.cc:579] Killed test thread 2/10
I20250111 02:57:54.420682 18513 master-stress-test.cc:577] Killing test thread 3/10
I20250111 02:57:54.420984 18513 master-stress-test.cc:579] Killed test thread 3/10
I20250111 02:57:54.421237 18513 master-stress-test.cc:577] Killing test thread 4/10
I20250111 02:57:54.421527 18513 master-stress-test.cc:579] Killed test thread 4/10
I20250111 02:57:54.421768 18513 master-stress-test.cc:577] Killing test thread 5/10
I20250111 02:57:54.430982 20707 tablet_replica.cc:331] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:54.432008 18624 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet d6da575f052d4b20a8e41f0bd9dab7e4
I20250111 02:57:54.433401 18624 catalog_manager.cc:6596] ReplaceTablet: tablet d6da575f052d4b20a8e41f0bd9dab7e4 deleted and replaced by tablet 5e03d4ebb40746e484b0fb4f4b2dd35d
I20250111 02:57:54.437927 19165 tablet_service.cc:1514] Processing DeleteTablet for tablet d6da575f052d4b20a8e41f0bd9dab7e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e03d4ebb40746e484b0fb4f4b2dd35d) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:54.438868 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet d6da575f052d4b20a8e41f0bd9dab7e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e03d4ebb40746e484b0fb4f4b2dd35d) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:54.444140 20709 tablet_replica.cc:331] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:54.446792 20708 tablet_replica.cc:331] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:54.447392 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet d6da575f052d4b20a8e41f0bd9dab7e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e03d4ebb40746e484b0fb4f4b2dd35d) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:54.447793 20708 raft_consensus.cc:2238] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:54.448516 20708 raft_consensus.cc:2267] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:54.460558 20707 raft_consensus.cc:2238] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 3 LEADER]: Raft consensus shutting down.
I20250111 02:57:54.461686 20707 raft_consensus.cc:2267] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:54.464090 20709 raft_consensus.cc:2238] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:54.479936 20709 raft_consensus.cc:2267] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:54.482896 20709 ts_tablet_manager.cc:1905] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:54.486403 20707 ts_tablet_manager.cc:1905] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:54.450868 20708 ts_tablet_manager.cc:1905] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:54.566563 20708 ts_tablet_manager.cc:1918] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.2
I20250111 02:57:54.567152 20708 log.cc:1198] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/1ef92ac716824107916a1ea4fd49d5aa
I20250111 02:57:54.576211 20707 ts_tablet_manager.cc:1918] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.2
I20250111 02:57:54.578122 20707 log.cc:1198] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/1ef92ac716824107916a1ea4fd49d5aa
I20250111 02:57:54.579279 20707 ts_tablet_manager.cc:1939] T 1ef92ac716824107916a1ea4fd49d5aa P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:54.530464 20709 ts_tablet_manager.cc:1918] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.2
I20250111 02:57:54.579725 20708 ts_tablet_manager.cc:1939] T 1ef92ac716824107916a1ea4fd49d5aa P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:54.580192 20709 log.cc:1198] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/1ef92ac716824107916a1ea4fd49d5aa
I20250111 02:57:54.581691 20709 ts_tablet_manager.cc:1939] T 1ef92ac716824107916a1ea4fd49d5aa P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:54.583985 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet 1ef92ac716824107916a1ea4fd49d5aa (table default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609]) successfully deleted
I20250111 02:57:54.588757 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet 1ef92ac716824107916a1ea4fd49d5aa (table default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609]) successfully deleted
I20250111 02:57:54.592389 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 1ef92ac716824107916a1ea4fd49d5aa (table default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609]) successfully deleted
I20250111 02:57:54.593521 20709 tablet_replica.cc:331] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:54.594820 20709 raft_consensus.cc:2238] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:54.595551 20708 tablet_replica.cc:331] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f: stopping tablet replica
I20250111 02:57:54.595599 20709 raft_consensus.cc:2267] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:54.596387 20707 tablet_replica.cc:331] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c: stopping tablet replica
I20250111 02:57:54.597782 20708 raft_consensus.cc:2238] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:57:54.598573 20709 ts_tablet_manager.cc:1905] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:54.599401 20707 raft_consensus.cc:2238] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:54.600728 20708 raft_consensus.cc:2267] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:54.601485 20707 raft_consensus.cc:2267] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:54.605713 20708 ts_tablet_manager.cc:1905] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:54.612556 20707 ts_tablet_manager.cc:1905] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:54.638716 20709 ts_tablet_manager.cc:1918] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:54.639226 20709 log.cc:1198] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/d6da575f052d4b20a8e41f0bd9dab7e4
I20250111 02:57:54.640067 20709 ts_tablet_manager.cc:1939] T d6da575f052d4b20a8e41f0bd9dab7e4 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:54.643384 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet d6da575f052d4b20a8e41f0bd9dab7e4 (table default.Table_02b5991089a34b0b8ee6039ccc092c93 [id=977f59b134154bc396bf42bc1e44e717]) successfully deleted
I20250111 02:57:54.670581 20708 ts_tablet_manager.cc:1918] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:54.676127 20707 ts_tablet_manager.cc:1918] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:54.676733 20707 log.cc:1198] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/d6da575f052d4b20a8e41f0bd9dab7e4
I20250111 02:57:54.677198 20708 log.cc:1198] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/d6da575f052d4b20a8e41f0bd9dab7e4
I20250111 02:57:54.678558 20708 ts_tablet_manager.cc:1939] T d6da575f052d4b20a8e41f0bd9dab7e4 P 54628fbcec2f4359aa9e26a734d4d29f: Deleting consensus metadata
I20250111 02:57:54.679248 20707 ts_tablet_manager.cc:1939] T d6da575f052d4b20a8e41f0bd9dab7e4 P 8221604733fe4fc0a1fc32de8d6af71c: Deleting consensus metadata
I20250111 02:57:54.682608 18617 catalog_manager.cc:4872] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): tablet d6da575f052d4b20a8e41f0bd9dab7e4 (table default.Table_02b5991089a34b0b8ee6039ccc092c93 [id=977f59b134154bc396bf42bc1e44e717]) successfully deleted
I20250111 02:57:54.692073 18615 catalog_manager.cc:4872] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623): tablet d6da575f052d4b20a8e41f0bd9dab7e4 (table default.Table_02b5991089a34b0b8ee6039ccc092c93 [id=977f59b134154bc396bf42bc1e44e717]) successfully deleted
I20250111 02:57:54.768460 19173 tablet_service.cc:1467] Processing CreateTablet for tablet fb9b1c7e252d40e29e6d16311d145968 (DEFAULT_TABLE table=default.Table_e091dc5f47d9492aa86ff883e04fe415 [id=ee752760552944289c461f1c4c50d90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.770393 19173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb9b1c7e252d40e29e6d16311d145968. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.775975 19034 tablet_service.cc:1467] Processing CreateTablet for tablet fb9b1c7e252d40e29e6d16311d145968 (DEFAULT_TABLE table=default.Table_e091dc5f47d9492aa86ff883e04fe415 [id=ee752760552944289c461f1c4c50d90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.777743 19034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb9b1c7e252d40e29e6d16311d145968. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.779744 18881 tablet_service.cc:1467] Processing CreateTablet for tablet fb9b1c7e252d40e29e6d16311d145968 (DEFAULT_TABLE table=default.Table_e091dc5f47d9492aa86ff883e04fe415 [id=ee752760552944289c461f1c4c50d90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.781452 18881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb9b1c7e252d40e29e6d16311d145968. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.790088 19165 tablet_service.cc:1467] Processing CreateTablet for tablet 16d88c5da4f345c3a43a1ef00f82856b (DEFAULT_TABLE table=default.Table_a7b2b5af5dfb439a84159b75de0d03b1 [id=dc7854580f744d4083f6aa3f0f0f7bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.791939 19165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16d88c5da4f345c3a43a1ef00f82856b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.797629 19033 tablet_service.cc:1467] Processing CreateTablet for tablet 16d88c5da4f345c3a43a1ef00f82856b (DEFAULT_TABLE table=default.Table_a7b2b5af5dfb439a84159b75de0d03b1 [id=dc7854580f744d4083f6aa3f0f0f7bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.798817 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16d88c5da4f345c3a43a1ef00f82856b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.816152 18630 catalog_manager.cc:5526] T 7145efe5c0094b70be2cd6481b770248 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:54.818240 18630 catalog_manager.cc:5526] T b2f561cfd4f34426b7e42027e53b3ff1 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:54.818600 20718 tablet_bootstrap.cc:492] T fb9b1c7e252d40e29e6d16311d145968 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
I20250111 02:57:54.825410 18885 tablet_service.cc:1467] Processing CreateTablet for tablet 16d88c5da4f345c3a43a1ef00f82856b (DEFAULT_TABLE table=default.Table_a7b2b5af5dfb439a84159b75de0d03b1 [id=dc7854580f744d4083f6aa3f0f0f7bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.826081 19171 tablet_service.cc:1467] Processing CreateTablet for tablet 0255034337f24337be838425136d8b4c (DEFAULT_TABLE table=default.Table_813d4e8bb0f34a46bf96ed3bee6ca86f [id=d790288a097a4416b68b93db6ac847fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.827028 18885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16d88c5da4f345c3a43a1ef00f82856b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.830672 18630 catalog_manager.cc:5526] T aca35147034446b3882b5244923bddc7 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:54.833886 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 0255034337f24337be838425136d8b4c (DEFAULT_TABLE table=default.Table_813d4e8bb0f34a46bf96ed3bee6ca86f [id=d790288a097a4416b68b93db6ac847fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.834378 18630 catalog_manager.cc:5526] T fa86b1e6eca7403c9a9c52592b8fb636 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:54.836549 18884 tablet_service.cc:1467] Processing CreateTablet for tablet 0d4ba33836f04ed19d67b8bc161e57c4 (DEFAULT_TABLE table=default.Table_bf5cf724093d4691a60379b87cfab2f0 [id=c151a212f267426a9cb6180c938a5815]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.837126 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet fa86b1e6eca7403c9a9c52592b8fb636 (table default.Table_4b9e85e01e1843ce831205177cbb2e0f [id=0a118da8c5d0408980cbd0b33e0c2ed0]). Expected version 1 got 0
I20250111 02:57:54.837905 19030 tablet_service.cc:1467] Processing CreateTablet for tablet 0d4ba33836f04ed19d67b8bc161e57c4 (DEFAULT_TABLE table=default.Table_bf5cf724093d4691a60379b87cfab2f0 [id=c151a212f267426a9cb6180c938a5815]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.838819 19168 tablet_service.cc:1467] Processing CreateTablet for tablet 0d4ba33836f04ed19d67b8bc161e57c4 (DEFAULT_TABLE table=default.Table_bf5cf724093d4691a60379b87cfab2f0 [id=c151a212f267426a9cb6180c938a5815]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.840008 19030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d4ba33836f04ed19d67b8bc161e57c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.840344 19168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d4ba33836f04ed19d67b8bc161e57c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.838132 18630 catalog_manager.cc:5526] T a6e4b222bf974a8d80b25141a5686761 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:54.842599 18630 catalog_manager.cc:5526] T d837bd001aa649e9b3ca94f7cdd9f0fe P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:54.843884 19171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0255034337f24337be838425136d8b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.844095 19034 tablet_service.cc:1467] Processing CreateTablet for tablet 0255034337f24337be838425136d8b4c (DEFAULT_TABLE table=default.Table_813d4e8bb0f34a46bf96ed3bee6ca86f [id=d790288a097a4416b68b93db6ac847fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.845494 19034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0255034337f24337be838425136d8b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.845779 18878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0255034337f24337be838425136d8b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.851604 18884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d4ba33836f04ed19d67b8bc161e57c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.855945 18630 catalog_manager.cc:5526] T 0b6687f39d4246b49c0c3eb904b11c1a P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65). New cstate: current_term: 1 leader_uuid: "8221604733fe4fc0a1fc32de8d6af71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } }
I20250111 02:57:54.858199 18882 tablet_service.cc:1467] Processing CreateTablet for tablet 12065effb54b4d4a94349641109b578c (DEFAULT_TABLE table=default.Table_d7e00ace61e545129a2e99093dba16db [id=d0e7184505c24fc5b9960e8b6d1c955d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.859472 18882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12065effb54b4d4a94349641109b578c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.857544 18630 catalog_manager.cc:5526] T 2009d6f54e2842e8be3af2cc9147dc2c P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:54.860921 18630 catalog_manager.cc:5526] T 1b69a8c2caea4bddac3f539733bcd3af P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:54.864213 18630 catalog_manager.cc:5526] T cb96635f30b34de48870daf5166813bc P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65). New cstate: current_term: 1 leader_uuid: "8221604733fe4fc0a1fc32de8d6af71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } }
I20250111 02:57:54.866061 18630 catalog_manager.cc:5526] T 8414653def014ce98bc38d87aac75bae P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:54.867825 18630 catalog_manager.cc:5526] T b4273afc264843029ed0baf1fc5c8a3e P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65). New cstate: current_term: 1 leader_uuid: "8221604733fe4fc0a1fc32de8d6af71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } }
I20250111 02:57:54.876142 18886 tablet_service.cc:1467] Processing CreateTablet for tablet 788c34eff1884d979add7e09615c02ae (DEFAULT_TABLE table=default.Table_dfe42fac0a764052a8d2f6add8882b64 [id=b42908f2a2c14d0a84027cb7a1d775b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.877422 18886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 788c34eff1884d979add7e09615c02ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.879709 19028 tablet_service.cc:1467] Processing CreateTablet for tablet 12065effb54b4d4a94349641109b578c (DEFAULT_TABLE table=default.Table_d7e00ace61e545129a2e99093dba16db [id=d0e7184505c24fc5b9960e8b6d1c955d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.881177 19028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12065effb54b4d4a94349641109b578c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.884815 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 12065effb54b4d4a94349641109b578c (DEFAULT_TABLE table=default.Table_d7e00ace61e545129a2e99093dba16db [id=d0e7184505c24fc5b9960e8b6d1c955d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.886042 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12065effb54b4d4a94349641109b578c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.916268 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet b4273afc264843029ed0baf1fc5c8a3e (table default.Table_4e3a79a5d2184687a8c51fd2702c5fd0 [id=73c59c0d9864497fbc9f19e265d8283f]). Expected version 1 got 0
I20250111 02:57:54.922088 18630 catalog_manager.cc:5526] T 3514d795a0f24adcaa2accc0cf05b142 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:54.927697 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet 3514d795a0f24adcaa2accc0cf05b142 (table default.Table_4c9aebb06d554107a5448d968fdb0436 [id=3fab00618e6548de85b5d20adee4e9ac]). Expected version 1 got 0
I20250111 02:57:54.930014 18630 catalog_manager.cc:5526] T 237ea33846044127987c1d81a3203553 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:54.938051 19026 tablet_service.cc:1467] Processing CreateTablet for tablet 788c34eff1884d979add7e09615c02ae (DEFAULT_TABLE table=default.Table_dfe42fac0a764052a8d2f6add8882b64 [id=b42908f2a2c14d0a84027cb7a1d775b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.941308 20718 tablet_bootstrap.cc:654] T fb9b1c7e252d40e29e6d16311d145968 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:54.936551 18630 catalog_manager.cc:5526] T 6a03880a8b0e46b7858ed1c2c2dddcaa P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:54.962402 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]). Expected version 4 got 0
I20250111 02:57:54.962427 18956 heartbeater.cc:502] Master 127.18.20.125:40359 was elected leader, sending a full tablet report...
I20250111 02:57:54.967088 18630 catalog_manager.cc:5526] T 855dac47872d4c5fbf8f5e38b588c7fe P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:54.972327 18879 tablet_service.cc:1467] Processing CreateTablet for tablet c8f0c42e06d4428284fa006fe0062cf0 (DEFAULT_TABLE table=default.Table_6ad0b827920c42c68113310190019df1 [id=b25314ef99fc4e0ba80c6391223d2a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.973423 18879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8f0c42e06d4428284fa006fe0062cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:54.975410 19243 heartbeater.cc:502] Master 127.18.20.125:40359 was elected leader, sending a full tablet report...
I20250111 02:57:54.976830 19169 tablet_service.cc:1467] Processing CreateTablet for tablet 788c34eff1884d979add7e09615c02ae (DEFAULT_TABLE table=default.Table_dfe42fac0a764052a8d2f6add8882b64 [id=b42908f2a2c14d0a84027cb7a1d775b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:54.974831 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet 855dac47872d4c5fbf8f5e38b588c7fe (table default.Table_49e4dd58d29f4cb6a0ed4f85138713b8 [id=f1373922978a4598b888e1863525942e]). Expected version 1 got 0
I20250111 02:57:54.984684 18630 catalog_manager.cc:5526] T d3b7616fe56f4e5fb49bdde6961fa008 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65). New cstate: current_term: 1 leader_uuid: "8221604733fe4fc0a1fc32de8d6af71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } }
I20250111 02:57:55.000367 19023 tablet_service.cc:1467] Processing CreateTablet for tablet c8f0c42e06d4428284fa006fe0062cf0 (DEFAULT_TABLE table=default.Table_6ad0b827920c42c68113310190019df1 [id=b25314ef99fc4e0ba80c6391223d2a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.003748 18890 tablet_service.cc:1467] Processing CreateTablet for tablet f9092a3453f94abbbf4bdd66a57d42cc (DEFAULT_TABLE table=default.Table_ee18021e6a544ed6b86b624aec8c39be [id=a9538f81833c4c0d8d02b6271a67f711]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.005867 19170 tablet_service.cc:1467] Processing CreateTablet for tablet c8f0c42e06d4428284fa006fe0062cf0 (DEFAULT_TABLE table=default.Table_6ad0b827920c42c68113310190019df1 [id=b25314ef99fc4e0ba80c6391223d2a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.006572 19031 tablet_service.cc:1467] Processing CreateTablet for tablet f9092a3453f94abbbf4bdd66a57d42cc (DEFAULT_TABLE table=default.Table_ee18021e6a544ed6b86b624aec8c39be [id=a9538f81833c4c0d8d02b6271a67f711]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.012128 19177 tablet_service.cc:1467] Processing CreateTablet for tablet f9092a3453f94abbbf4bdd66a57d42cc (DEFAULT_TABLE table=default.Table_ee18021e6a544ed6b86b624aec8c39be [id=a9538f81833c4c0d8d02b6271a67f711]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.016616 18880 tablet_service.cc:1467] Processing CreateTablet for tablet 94bb9089c71b49bea9c0f4d94fc22fb1 (DEFAULT_TABLE table=default.Table_b19a186d285d4b5b845727344f7d155b [id=b28b429fe5844cd6ac6f4de349c9449b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.018450 19029 tablet_service.cc:1467] Processing CreateTablet for tablet 94bb9089c71b49bea9c0f4d94fc22fb1 (DEFAULT_TABLE table=default.Table_b19a186d285d4b5b845727344f7d155b [id=b28b429fe5844cd6ac6f4de349c9449b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.019662 19177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9092a3453f94abbbf4bdd66a57d42cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.020275 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 788c34eff1884d979add7e09615c02ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.026132 18887 tablet_service.cc:1467] Processing CreateTablet for tablet f9ec6b2a27744ee1aef0efdc98d8757c (DEFAULT_TABLE table=default.Table_f9d19027fe814825a6c3906baef77ff3 [id=9a4fbc1c2c2c44c182f9935156885c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.029116 19027 tablet_service.cc:1467] Processing CreateTablet for tablet f9ec6b2a27744ee1aef0efdc98d8757c (DEFAULT_TABLE table=default.Table_f9d19027fe814825a6c3906baef77ff3 [id=9a4fbc1c2c2c44c182f9935156885c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.032366 19170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8f0c42e06d4428284fa006fe0062cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.032213 19024 tablet_service.cc:1467] Processing CreateTablet for tablet 63aef98c83d94918bb3163d491870084 (DEFAULT_TABLE table=default.Table_98958a28faeb43e8a3b34fcf20d936aa [id=98e24956666a413cbf4564438a594ef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.005640 18630 catalog_manager.cc:5526] T 845da3ee0863430ea3fe9de8c84f2c17 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.036038 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet 845da3ee0863430ea3fe9de8c84f2c17 (table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3]). Expected version 2 got 0
I20250111 02:57:55.035862 18889 tablet_service.cc:1467] Processing CreateTablet for tablet 63aef98c83d94918bb3163d491870084 (DEFAULT_TABLE table=default.Table_98958a28faeb43e8a3b34fcf20d936aa [id=98e24956666a413cbf4564438a594ef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.037025 18630 catalog_manager.cc:5526] T 0158a8849bf44a4fbb5daca22883478f P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.037554 18888 tablet_service.cc:1467] Processing CreateTablet for tablet 315ac86194b04802909a573876fce64b (DEFAULT_TABLE table=default.Table_ea068446b5434b3ca5f0afa8386c0655 [id=dc09e842add34ef099edfd1296255c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.044730 19022 tablet_service.cc:1467] Processing CreateTablet for tablet 315ac86194b04802909a573876fce64b (DEFAULT_TABLE table=default.Table_ea068446b5434b3ca5f0afa8386c0655 [id=dc09e842add34ef099edfd1296255c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.046826 20722 tablet_bootstrap.cc:492] T fb9b1c7e252d40e29e6d16311d145968 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:55.059428 18883 tablet_service.cc:1467] Processing CreateTablet for tablet dea4db154779483cb069118e3f3789b0 (DEFAULT_TABLE table=default.Table_8eae98e1e49c4f6c9e52644256812eaa [id=7dfae414add74b6491f2f7595a551342]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.061033 19025 tablet_service.cc:1467] Processing CreateTablet for tablet dea4db154779483cb069118e3f3789b0 (DEFAULT_TABLE table=default.Table_8eae98e1e49c4f6c9e52644256812eaa [id=7dfae414add74b6491f2f7595a551342]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.061139 19174 tablet_service.cc:1467] Processing CreateTablet for tablet f9ec6b2a27744ee1aef0efdc98d8757c (DEFAULT_TABLE table=default.Table_f9d19027fe814825a6c3906baef77ff3 [id=9a4fbc1c2c2c44c182f9935156885c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.063730 19167 tablet_service.cc:1467] Processing CreateTablet for tablet 63aef98c83d94918bb3163d491870084 (DEFAULT_TABLE table=default.Table_98958a28faeb43e8a3b34fcf20d936aa [id=98e24956666a413cbf4564438a594ef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.065771 19166 tablet_service.cc:1467] Processing CreateTablet for tablet 315ac86194b04802909a573876fce64b (DEFAULT_TABLE table=default.Table_ea068446b5434b3ca5f0afa8386c0655 [id=dc09e842add34ef099edfd1296255c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.068657 19172 tablet_service.cc:1467] Processing CreateTablet for tablet dea4db154779483cb069118e3f3789b0 (DEFAULT_TABLE table=default.Table_8eae98e1e49c4f6c9e52644256812eaa [id=7dfae414add74b6491f2f7595a551342]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.072424 18877 tablet_service.cc:1467] Processing CreateTablet for tablet d2fc07e0474e4fc5a46f2350229ca31f (DEFAULT_TABLE table=default.Table_6a0ed66485f640e9a2c11b62b5cbd239 [id=7b936eef4300409e908cc92b1453b500]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.073983 19032 tablet_service.cc:1467] Processing CreateTablet for tablet d2fc07e0474e4fc5a46f2350229ca31f (DEFAULT_TABLE table=default.Table_6a0ed66485f640e9a2c11b62b5cbd239 [id=7b936eef4300409e908cc92b1453b500]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.066093 19173 tablet_service.cc:1467] Processing CreateTablet for tablet 94bb9089c71b49bea9c0f4d94fc22fb1 (DEFAULT_TABLE table=default.Table_b19a186d285d4b5b845727344f7d155b [id=b28b429fe5844cd6ac6f4de349c9449b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.076135 18876 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3c5b038d994d07abfea701194e8ad9 (DEFAULT_TABLE table=default.Table_bdf982e1d6904892ac08ff16ed20a636 [id=a2b6467b58754b85ab0481a9e8aa0ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.077090 19021 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3c5b038d994d07abfea701194e8ad9 (DEFAULT_TABLE table=default.Table_bdf982e1d6904892ac08ff16ed20a636 [id=a2b6467b58754b85ab0481a9e8aa0ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.077327 19100 heartbeater.cc:502] Master 127.18.20.125:40359 was elected leader, sending a full tablet report...
I20250111 02:57:55.079813 18875 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee884f4d31f46c197c99bf72f1f19bf (DEFAULT_TABLE table=default.Table_85a0376bcaf24d0081e780e8c19764b8 [id=5f97e9e0988848319a9c8aaca6aad982]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.081490 19020 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee884f4d31f46c197c99bf72f1f19bf (DEFAULT_TABLE table=default.Table_85a0376bcaf24d0081e780e8c19764b8 [id=5f97e9e0988848319a9c8aaca6aad982]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.084780 18874 tablet_service.cc:1467] Processing CreateTablet for tablet aec6dffdaa6b4798b7827a6a445601ca (DEFAULT_TABLE table=default.Table_0dcad07d56d843e59f351d933e625a21 [id=57e820dd99764a578d0c50f5f250aad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.085897 19019 tablet_service.cc:1467] Processing CreateTablet for tablet aec6dffdaa6b4798b7827a6a445601ca (DEFAULT_TABLE table=default.Table_0dcad07d56d843e59f351d933e625a21 [id=57e820dd99764a578d0c50f5f250aad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.088055 18630 catalog_manager.cc:5526] T 933828fd6fb041e9a4889f4ea8a3c5ff P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to d0cedab82b154d62b78809edc6f28bfa (127.18.20.67). New cstate: current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } }
I20250111 02:57:55.090481 18890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9092a3453f94abbbf4bdd66a57d42cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.091276 18873 tablet_service.cc:1467] Processing CreateTablet for tablet bc0bd71c6f4d4c90a4f3ea747af02165 (DEFAULT_TABLE table=default.Table_535081a0b0ff46998f523371377b75f9 [id=531ca04d38e9427eb955326135436b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.092453 18873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0bd71c6f4d4c90a4f3ea747af02165. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.092900 19018 tablet_service.cc:1467] Processing CreateTablet for tablet bc0bd71c6f4d4c90a4f3ea747af02165 (DEFAULT_TABLE table=default.Table_535081a0b0ff46998f523371377b75f9 [id=531ca04d38e9427eb955326135436b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.094784 18880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94bb9089c71b49bea9c0f4d94fc22fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.099396 18887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9ec6b2a27744ee1aef0efdc98d8757c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.100054 18883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dea4db154779483cb069118e3f3789b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.100775 18877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2fc07e0474e4fc5a46f2350229ca31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.101354 18889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63aef98c83d94918bb3163d491870084. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.103253 18874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aec6dffdaa6b4798b7827a6a445601ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.107532 18888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 315ac86194b04802909a573876fce64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.111497 18876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3c5b038d994d07abfea701194e8ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.115514 18875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee884f4d31f46c197c99bf72f1f19bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.107713 18630 catalog_manager.cc:5526] T ffd063f8889b4bcdb9656466f0b144f3 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.120499 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet ffd063f8889b4bcdb9656466f0b144f3 (table default.Table_53d44be8eaeb4e59a8e3454aac56daca [id=8de61815471842e593a7d29efffc443c]). Expected version 1 got 0
I20250111 02:57:55.121326 18630 catalog_manager.cc:5526] T e46894710a72407993b36801f3b90eb0 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to d0cedab82b154d62b78809edc6f28bfa (127.18.20.67). New cstate: current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } }
I20250111 02:57:55.124816 19174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9ec6b2a27744ee1aef0efdc98d8757c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.132213 19167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63aef98c83d94918bb3163d491870084. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.135689 19166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 315ac86194b04802909a573876fce64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.135789 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet e46894710a72407993b36801f3b90eb0 (table default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3]). Expected version 2 got 0
I20250111 02:57:55.136845 18630 catalog_manager.cc:5526] T 220da826a0f140d59193e8ad72096b60 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.139743 20722 tablet_bootstrap.cc:654] T fb9b1c7e252d40e29e6d16311d145968 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:55.145687 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet 220da826a0f140d59193e8ad72096b60 (table default.Table_5e7c139097914dd7b7c8e55058d5290d [id=9143d1d3b8754ac78d615a1cc4601d86]). Expected version 2 got 0
I20250111 02:57:55.143676 19172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dea4db154779483cb069118e3f3789b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.147374 19173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94bb9089c71b49bea9c0f4d94fc22fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.165017 18630 catalog_manager.cc:5526] T ea76b354a75041e888cf862250e4947b P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.189472 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet ea76b354a75041e888cf862250e4947b (table default.Table_b44f1ebbaf7045e2bcfefcee1c91c0d9 [id=ec5cf5f9b6234008914c0ed664260dcb]). Expected version 1 got 0
I20250111 02:57:55.192749 20722 tablet_bootstrap.cc:492] T fb9b1c7e252d40e29e6d16311d145968 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:55.202891 19177 tablet_service.cc:1467] Processing CreateTablet for tablet d2fc07e0474e4fc5a46f2350229ca31f (DEFAULT_TABLE table=default.Table_6a0ed66485f640e9a2c11b62b5cbd239 [id=7b936eef4300409e908cc92b1453b500]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.208678 19177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2fc07e0474e4fc5a46f2350229ca31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.215560 20722 ts_tablet_manager.cc:1397] T fb9b1c7e252d40e29e6d16311d145968 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.169s user 0.010s sys 0.003s
I20250111 02:57:55.194178 18630 catalog_manager.cc:5526] T e4bd42ad66064bcaa043746081f1c759 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65). New cstate: current_term: 1 leader_uuid: "8221604733fe4fc0a1fc32de8d6af71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } }
I20250111 02:57:55.216408 18630 catalog_manager.cc:5618] TS 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623) does not have the latest schema for tablet e4bd42ad66064bcaa043746081f1c759 (table default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7]). Expected version 2 got 0
I20250111 02:57:55.223412 18630 catalog_manager.cc:5526] T adf61f6cb63d4e8ea46b440e7ecce00d P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.229485 18630 catalog_manager.cc:5526] T aaab428f798a44f69c80e5792d56b920 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.237533 20722 raft_consensus.cc:357] T fb9b1c7e252d40e29e6d16311d145968 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
W20250111 02:57:55.239194 18986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.20.67:35591, user_credentials={real_user=slave}} blocked reactor thread for 44742.9us
I20250111 02:57:55.240399 18630 catalog_manager.cc:5526] T e01771c56ea5498a876d3bb865c752a2 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.245294 18885 tablet_service.cc:1467] Processing CreateTablet for tablet 900d4d665c4a4048bc6989ba63391cc1 (DEFAULT_TABLE table=default.Table_baa2b0022e914aa5b485db90037be4af [id=3f31cb2959f340e3ae2656b63eaa52e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.246892 18885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 900d4d665c4a4048bc6989ba63391cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.246765 20722 raft_consensus.cc:383] T fb9b1c7e252d40e29e6d16311d145968 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:55.247277 20722 raft_consensus.cc:738] T fb9b1c7e252d40e29e6d16311d145968 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:55.249601 20722 consensus_queue.cc:260] T fb9b1c7e252d40e29e6d16311d145968 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:55.248454 18630 catalog_manager.cc:5526] T 2f11541b436e42b7b0728da2aaec48f7 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.248013 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 29f27ed2c53d43d7a51ee2e45afaf9fc (DEFAULT_TABLE table=default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.252938 18878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f27ed2c53d43d7a51ee2e45afaf9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.256114 20722 ts_tablet_manager.cc:1428] T fb9b1c7e252d40e29e6d16311d145968 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.035s user 0.007s sys 0.001s
I20250111 02:57:55.254005 18630 catalog_manager.cc:5526] T 6ff06134303d4cb0b4292fe1c0c2d443 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.262554 18630 catalog_manager.cc:5526] T 3c57d85390324bbc9ea30a709faec01c P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.264258 18630 catalog_manager.cc:5526] T 934c900baf1b4000af7d7ec60cbb3591 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.264719 19175 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3c5b038d994d07abfea701194e8ad9 (DEFAULT_TABLE table=default.Table_bdf982e1d6904892ac08ff16ed20a636 [id=a2b6467b58754b85ab0481a9e8aa0ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.265952 19175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3c5b038d994d07abfea701194e8ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.267712 20692 raft_consensus.cc:491] T fb9b1c7e252d40e29e6d16311d145968 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:55.268313 20692 raft_consensus.cc:513] T fb9b1c7e252d40e29e6d16311d145968 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:55.272001 20692 leader_election.cc:290] T fb9b1c7e252d40e29e6d16311d145968 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623), 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535)
I20250111 02:57:55.266824 18630 catalog_manager.cc:5526] T df1870cceafe463b8d024e8b065850e0 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.280218 18630 catalog_manager.cc:5526] T 56fb0bf949904208af003f08caae7594 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.282016 18630 catalog_manager.cc:5526] T f31ad2e1895d4a26b8e9f7cd7366ef81 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } }
W20250111 02:57:55.286056 19133 leader_election.cc:343] T fb9b1c7e252d40e29e6d16311d145968 P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Not found: Tablet not found: fb9b1c7e252d40e29e6d16311d145968
I20250111 02:57:55.288136 19172 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee884f4d31f46c197c99bf72f1f19bf (DEFAULT_TABLE table=default.Table_85a0376bcaf24d0081e780e8c19764b8 [id=5f97e9e0988848319a9c8aaca6aad982]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.288666 19164 tablet_service.cc:1467] Processing CreateTablet for tablet aec6dffdaa6b4798b7827a6a445601ca (DEFAULT_TABLE table=default.Table_0dcad07d56d843e59f351d933e625a21 [id=57e820dd99764a578d0c50f5f250aad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.289294 19172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee884f4d31f46c197c99bf72f1f19bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.289857 19164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aec6dffdaa6b4798b7827a6a445601ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.287297 18630 catalog_manager.cc:5526] T 0c669a3ff8aa46fa8b04b86ed3bfde49 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.307988 18630 catalog_manager.cc:5526] T 64abb30da7fc452cad774bbe59b498f4 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to d0cedab82b154d62b78809edc6f28bfa (127.18.20.67). New cstate: current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } }
I20250111 02:57:55.310273 18630 catalog_manager.cc:5526] T 36f71f8dd2894d40b3bfdd20c1453787 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to d0cedab82b154d62b78809edc6f28bfa (127.18.20.67). New cstate: current_term: 1 leader_uuid: "d0cedab82b154d62b78809edc6f28bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } }
I20250111 02:57:55.314667 18630 catalog_manager.cc:5526] T 8eb87813c28742edad99470c84269ef4 P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.317440 19163 tablet_service.cc:1467] Processing CreateTablet for tablet bc0bd71c6f4d4c90a4f3ea747af02165 (DEFAULT_TABLE table=default.Table_535081a0b0ff46998f523371377b75f9 [id=531ca04d38e9427eb955326135436b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.319279 19163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0bd71c6f4d4c90a4f3ea747af02165. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.318872 18630 catalog_manager.cc:5526] T 79f4500d3beb4c9bb20f84c661c3758b P 54628fbcec2f4359aa9e26a734d4d29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66). New cstate: current_term: 1 leader_uuid: "54628fbcec2f4359aa9e26a734d4d29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } health_report { overall_health: HEALTHY } } }
I20250111 02:57:55.325350 18628 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet 07f15096ea484b499fbdecec74eb6f66
I20250111 02:57:55.325718 18627 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: Sending DeleteTablet for 3 replicas of tablet d83cfd85a52f438dabe49b6223d5090f
I20250111 02:57:55.326938 18628 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47522 (request call id 318) took 1295 ms. Trace:
I20250111 02:57:55.327112 18627 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47522 (request call id 317) took 1296 ms. Trace:
I20250111 02:57:55.327383 18627 rpcz_store.cc:276] 0111 02:57:54.031186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:54.031426 (+ 240us) service_pool.cc:225] Handling call
0111 02:57:54.033346 (+ 1920us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0111 02:57:54.039673 (+ 6327us) catalog_manager.cc:2753] Modifying in-memory table state
0111 02:57:54.039890 (+ 217us) catalog_manager.cc:2762] Locking tablets
0111 02:57:54.040664 (+ 774us) catalog_manager.cc:2776] Removing table and tablets from system table
0111 02:57:54.433997 (+393333us) catalog_manager.cc:2794] Removing table from by-name map
0111 02:57:54.434215 (+ 218us) catalog_manager.cc:2810] Committing in-memory state
0111 02:57:55.325635 (+891420us) catalog_manager.cc:2814] Aborting table tasks
0111 02:57:55.327023 (+ 1388us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:54.138235 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:54.139771 (+ 1536us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:54.139796 (+ 25us) write_op.cc:435] Acquired schema lock
0111 02:57:54.139812 (+ 16us) tablet.cc:659] Decoding operations
0111 02:57:54.140177 (+ 365us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:54.140218 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:54.140243 (+ 25us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:54.140557 (+ 314us) tablet.cc:698] Row locks acquired
0111 02:57:54.140576 (+ 19us) write_op.cc:260] PREPARE: finished
0111 02:57:54.140707 (+ 131us) write_op.cc:270] Start()
0111 02:57:54.140797 (+ 90us) write_op.cc:276] Timestamp: P: 1736564274140671 usec, L: 0
0111 02:57:54.140816 (+ 19us) op_driver.cc:348] REPLICATION: starting
0111 02:57:54.141382 (+ 566us) log.cc:844] Serialized 728 byte log entry
0111 02:57:54.427963 (+286581us) op_driver.cc:464] REPLICATION: finished
0111 02:57:54.428239 (+ 276us) write_op.cc:301] APPLY: starting
0111 02:57:54.428299 (+ 60us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:54.428403 (+ 104us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:54.428496 (+ 93us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:54.428826 (+ 330us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:54.429003 (+ 177us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0111 02:57:54.429062 (+ 59us) write_op.cc:312] APPLY: finished
0111 02:57:54.429248 (+ 186us) log.cc:844] Serialized 29 byte log entry
0111 02:57:54.429406 (+ 158us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:54.429494 (+ 88us) write_op.cc:454] Released schema lock
0111 02:57:54.429535 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:54.429800 (+ 265us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:54.429824 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":1514,"apply.run_wall_time_us":1514,"num_ops":2,"prepare.queue_time_us":96011,"prepare.run_cpu_time_us":2524,"prepare.run_wall_time_us":4632,"replication_time_us":287044,"spinlock_wait_cycles":32512}]]}
I20250111 02:57:55.331662 18629 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47522 (request call id 316) took 1312 ms. Trace:
I20250111 02:57:55.332012 18629 rpcz_store.cc:276] 0111 02:57:54.019579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:54.025316 (+ 5737us) service_pool.cc:225] Handling call
0111 02:57:54.027228 (+ 1912us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0111 02:57:54.028418 (+ 1190us) catalog_manager.cc:3627] Apply alter schema
0111 02:57:54.029209 (+ 791us) catalog_manager.cc:3659] Acquired catalog manager lock
0111 02:57:54.029477 (+ 268us) catalog_manager.cc:3817] Updating metadata on disk
0111 02:57:54.409536 (+380059us) catalog_manager.cc:3848] Committing alterations to in-memory state
0111 02:57:55.331600 (+922064us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:54.119900 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:54.120500 (+ 600us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:54.120538 (+ 38us) write_op.cc:435] Acquired schema lock
0111 02:57:54.120562 (+ 24us) tablet.cc:659] Decoding operations
0111 02:57:54.120950 (+ 388us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:54.120996 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:54.121040 (+ 44us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:54.121246 (+ 206us) tablet.cc:698] Row locks acquired
0111 02:57:54.121280 (+ 34us) write_op.cc:260] PREPARE: finished
0111 02:57:54.121456 (+ 176us) write_op.cc:270] Start()
0111 02:57:54.121576 (+ 120us) write_op.cc:276] Timestamp: P: 1736564274121389 usec, L: 0
0111 02:57:54.121605 (+ 29us) op_driver.cc:348] REPLICATION: starting
0111 02:57:54.122361 (+ 756us) log.cc:844] Serialized 405 byte log entry
0111 02:57:54.404187 (+281826us) op_driver.cc:464] REPLICATION: finished
0111 02:57:54.405990 (+ 1803us) write_op.cc:301] APPLY: starting
0111 02:57:54.406078 (+ 88us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:54.406207 (+ 129us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:54.406357 (+ 150us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:54.406685 (+ 328us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:54.406865 (+ 180us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:54.406938 (+ 73us) write_op.cc:312] APPLY: finished
0111 02:57:54.407211 (+ 273us) log.cc:844] Serialized 23 byte log entry
0111 02:57:54.408073 (+ 862us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:54.408149 (+ 76us) write_op.cc:454] Released schema lock
0111 02:57:54.408188 (+ 39us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":3840,"child_traces":[["op",{"apply.queue_time_us":1569,"apply.run_cpu_time_us":1825,"apply.run_wall_time_us":2477,"num_ops":1,"prepare.queue_time_us":88925,"prepare.run_cpu_time_us":2736,"prepare.run_wall_time_us":2733,"replication_time_us":282427}]]}
I20250111 02:57:55.327288 18628 rpcz_store.cc:276] 0111 02:57:54.031558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:54.035199 (+ 3641us) service_pool.cc:225] Handling call
0111 02:57:54.064560 (+ 29361us) spinlock_profiling.cc:243] Waited 26 ms on lock 0x7b5400021554. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd427f31d50 00007fd427fb4f2a 00007fd429c59dec 00007fd429d53616 00007fd427abbb44 00007fd427abbad2 00007fd427abba01 00007fd427abb97c 00007fd427abac32 00007fd425d79dfc 00007fd425d79236
0111 02:57:54.066059 (+ 1499us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0111 02:57:54.066606 (+ 547us) catalog_manager.cc:2753] Modifying in-memory table state
0111 02:57:54.066761 (+ 155us) catalog_manager.cc:2762] Locking tablets
0111 02:57:54.075206 (+ 8445us) catalog_manager.cc:2776] Removing table and tablets from system table
0111 02:57:54.431493 (+356287us) catalog_manager.cc:2794] Removing table from by-name map
0111 02:57:54.431716 (+ 223us) catalog_manager.cc:2810] Committing in-memory state
0111 02:57:55.325249 (+893533us) catalog_manager.cc:2814] Aborting table tasks
0111 02:57:55.326843 (+ 1594us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:54.142995 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:54.143451 (+ 456us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:54.143482 (+ 31us) write_op.cc:435] Acquired schema lock
0111 02:57:54.143504 (+ 22us) tablet.cc:659] Decoding operations
0111 02:57:54.143855 (+ 351us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:54.143892 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:54.143921 (+ 29us) tablet.cc:682] Acquiring locks for 2 operations
0111 02:57:54.144205 (+ 284us) tablet.cc:698] Row locks acquired
0111 02:57:54.144223 (+ 18us) write_op.cc:260] PREPARE: finished
0111 02:57:54.144410 (+ 187us) write_op.cc:270] Start()
0111 02:57:54.144511 (+ 101us) write_op.cc:276] Timestamp: P: 1736564274144375 usec, L: 0
0111 02:57:54.144525 (+ 14us) op_driver.cc:348] REPLICATION: starting
0111 02:57:54.145075 (+ 550us) log.cc:844] Serialized 728 byte log entry
0111 02:57:54.428248 (+283173us) op_driver.cc:464] REPLICATION: finished
0111 02:57:54.430124 (+ 1876us) write_op.cc:301] APPLY: starting
0111 02:57:54.430178 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:54.430299 (+ 121us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:54.430384 (+ 85us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:54.430640 (+ 256us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:54.430783 (+ 143us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0111 02:57:54.430825 (+ 42us) write_op.cc:312] APPLY: finished
0111 02:57:54.431007 (+ 182us) log.cc:844] Serialized 29 byte log entry
0111 02:57:54.431162 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:54.431245 (+ 83us) write_op.cc:454] Released schema lock
0111 02:57:54.431276 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":59688320,"child_traces":[["op",{"apply.queue_time_us":1735,"apply.run_cpu_time_us":1345,"apply.run_wall_time_us":1344,"num_ops":2,"prepare.queue_time_us":37572,"prepare.run_cpu_time_us":2313,"prepare.run_wall_time_us":3470,"replication_time_us":283641,"thread_start_us":25656,"threads_started":1}]]}
I20250111 02:57:55.337680 18888 tablet_service.cc:1467] Processing CreateTablet for tablet 3d873a0faf4a4645a3969067ae0583dc (DEFAULT_TABLE table=default.Table_c9582d98645645f3aa80a65f7652cba8 [id=2d5d8f8a7ade443f91a7da34add25a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.339047 18888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d873a0faf4a4645a3969067ae0583dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.340536 18513 master-stress-test.cc:579] Killed test thread 5/10
I20250111 02:57:55.340476 18883 tablet_service.cc:1514] Processing DeleteTablet for tablet d83cfd85a52f438dabe49b6223d5090f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:54 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:55.340806 18513 master-stress-test.cc:577] Killing test thread 6/10
I20250111 02:57:55.341110 18513 master-stress-test.cc:579] Killed test thread 6/10
I20250111 02:57:55.341336 18513 master-stress-test.cc:577] Killing test thread 7/10
I20250111 02:57:55.341648 18513 master-stress-test.cc:579] Killed test thread 7/10
I20250111 02:57:55.341856 18513 master-stress-test.cc:577] Killing test thread 8/10
I20250111 02:57:55.341944 18876 tablet_service.cc:1514] Processing DeleteTablet for tablet 07f15096ea484b499fbdecec74eb6f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:54 UTC) from {username='slave'} at 127.0.0.1:51082
I20250111 02:57:55.342161 18513 master-stress-test.cc:579] Killed test thread 8/10
I20250111 02:57:55.342367 18513 master-stress-test.cc:577] Killing test thread 9/10
I20250111 02:57:55.342671 18513 master-stress-test.cc:579] Killed test thread 9/10
I20250111 02:57:55.342871 18513 master-stress-test.cc:577] Killing test thread 10/10
I20250111 02:57:55.343660 20722 tablet_bootstrap.cc:492] T f9092a3453f94abbbf4bdd66a57d42cc P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:55.343325 18875 tablet_service.cc:1467] Processing CreateTablet for tablet 8d20e8099b964fbc9e8e41447772d8d5 (DEFAULT_TABLE table=default.Table_6cdffa2e029846c3b95c8440c221426c [id=354a3a4b0b774fb9bc83623b2cb87f8a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250111 02:57:55.345422 18617 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 8d20e8099b964fbc9e8e41447772d8d5 (table default.Table_6cdffa2e029846c3b95c8440c221426c [id=354a3a4b0b774fb9bc83623b2cb87f8a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8d20e8099b964fbc9e8e41447772d8d5 consensus state has no leader
I20250111 02:57:55.348680 18513 master-stress-test.cc:579] Killed test thread 10/10
I20250111 02:57:55.349130 18513 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu with pid 18795
I20250111 02:57:55.349138 18875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d20e8099b964fbc9e8e41447772d8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.349433 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 900d4d665c4a4048bc6989ba63391cc1 (DEFAULT_TABLE table=default.Table_baa2b0022e914aa5b485db90037be4af [id=3f31cb2959f340e3ae2656b63eaa52e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.351840 20722 tablet_bootstrap.cc:654] T f9092a3453f94abbbf4bdd66a57d42cc P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:55.352391 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 900d4d665c4a4048bc6989ba63391cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.372066 20722 tablet_bootstrap.cc:492] T f9092a3453f94abbbf4bdd66a57d42cc P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:55.372750 20722 ts_tablet_manager.cc:1397] T f9092a3453f94abbbf4bdd66a57d42cc P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.009s
I20250111 02:57:55.375706 20722 raft_consensus.cc:357] T f9092a3453f94abbbf4bdd66a57d42cc P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:55.379977 20722 raft_consensus.cc:383] T f9092a3453f94abbbf4bdd66a57d42cc P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:55.380460 20722 raft_consensus.cc:738] T f9092a3453f94abbbf4bdd66a57d42cc P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:55.383874 19172 tablet_service.cc:1467] Processing CreateTablet for tablet 8d20e8099b964fbc9e8e41447772d8d5 (DEFAULT_TABLE table=default.Table_6cdffa2e029846c3b95c8440c221426c [id=354a3a4b0b774fb9bc83623b2cb87f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.385282 19177 tablet_service.cc:1467] Processing CreateTablet for tablet 218b87091d9546ba859f954a8f9e089e (DEFAULT_TABLE table=default.Table_c398ec562f594aca8c1e62cde38c04f9 [id=14bc02cbe30c486782d8fa4396855a9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.386018 19172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d20e8099b964fbc9e8e41447772d8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.386976 19177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 218b87091d9546ba859f954a8f9e089e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.388130 19169 tablet_service.cc:1467] Processing CreateTablet for tablet 29f27ed2c53d43d7a51ee2e45afaf9fc (DEFAULT_TABLE table=default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.389114 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f27ed2c53d43d7a51ee2e45afaf9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.392520 19165 tablet_service.cc:1514] Processing DeleteTablet for tablet 07f15096ea484b499fbdecec74eb6f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:54 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:55.381256 20722 consensus_queue.cc:260] T f9092a3453f94abbbf4bdd66a57d42cc P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:55.396684 19162 tablet_service.cc:1467] Processing CreateTablet for tablet 350f7c7f14004bb4bc3cb2338c7f172f (DEFAULT_TABLE table=default.Table_28d700d6f8444cfeab3dc52dd2bec17f [id=0e5eefce9dfa453eb277f75189def89e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.399819 19168 tablet_service.cc:1467] Processing CreateTablet for tablet a2eae136623e4802b083d6526c6867c9 (DEFAULT_TABLE table=default.Table_97e52a18fbd44f189d46ba9b26cd9033 [id=566e89e7552f4ce08fba16fe299b9ccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.400290 19162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 350f7c7f14004bb4bc3cb2338c7f172f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.408878 19168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2eae136623e4802b083d6526c6867c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.411821 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet d83cfd85a52f438dabe49b6223d5090f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:54 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:55.415092 19170 tablet_service.cc:1467] Processing CreateTablet for tablet 3d873a0faf4a4645a3969067ae0583dc (DEFAULT_TABLE table=default.Table_c9582d98645645f3aa80a65f7652cba8 [id=2d5d8f8a7ade443f91a7da34add25a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.416640 19170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d873a0faf4a4645a3969067ae0583dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.421959 19026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 788c34eff1884d979add7e09615c02ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.424392 19031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9092a3453f94abbbf4bdd66a57d42cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.427465 19019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aec6dffdaa6b4798b7827a6a445601ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.429261 19025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dea4db154779483cb069118e3f3789b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.429589 20718 tablet_bootstrap.cc:492] T fb9b1c7e252d40e29e6d16311d145968 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:55.430105 20718 ts_tablet_manager.cc:1397] T fb9b1c7e252d40e29e6d16311d145968 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.612s user 0.044s sys 0.027s
I20250111 02:57:55.432435 19029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94bb9089c71b49bea9c0f4d94fc22fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.436640 19021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3c5b038d994d07abfea701194e8ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.443944 20718 raft_consensus.cc:357] T fb9b1c7e252d40e29e6d16311d145968 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:55.445034 20718 raft_consensus.cc:383] T fb9b1c7e252d40e29e6d16311d145968 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:55.445341 20718 raft_consensus.cc:738] T fb9b1c7e252d40e29e6d16311d145968 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:55.455413 19020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee884f4d31f46c197c99bf72f1f19bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.455606 20718 consensus_queue.cc:260] T fb9b1c7e252d40e29e6d16311d145968 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } }
I20250111 02:57:55.462934 19022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 315ac86194b04802909a573876fce64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.484215 20718 ts_tablet_manager.cc:1428] T fb9b1c7e252d40e29e6d16311d145968 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.054s user 0.004s sys 0.007s
I20250111 02:57:55.485255 20718 tablet_bootstrap.cc:492] T 0d4ba33836f04ed19d67b8bc161e57c4 P 54628fbcec2f4359aa9e26a734d4d29f: Bootstrap starting.
W20250111 02:57:55.533097 19249 tablet.cc:2367] T fb9b1c7e252d40e29e6d16311d145968 P d0cedab82b154d62b78809edc6f28bfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:57:55.563872 19033 tablet_service.cc:1467] Processing CreateTablet for tablet 900d4d665c4a4048bc6989ba63391cc1 (DEFAULT_TABLE table=default.Table_baa2b0022e914aa5b485db90037be4af [id=3f31cb2959f340e3ae2656b63eaa52e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.562809 19032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2fc07e0474e4fc5a46f2350229ca31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.569917 19033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 900d4d665c4a4048bc6989ba63391cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.570528 19027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9ec6b2a27744ee1aef0efdc98d8757c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.574071 20718 tablet_bootstrap.cc:654] T 0d4ba33836f04ed19d67b8bc161e57c4 P 54628fbcec2f4359aa9e26a734d4d29f: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:55.575189 19023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8f0c42e06d4428284fa006fe0062cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.576640 20722 ts_tablet_manager.cc:1428] T f9092a3453f94abbbf4bdd66a57d42cc P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.204s user 0.004s sys 0.004s
I20250111 02:57:55.579417 20722 tablet_bootstrap.cc:492] T dea4db154779483cb069118e3f3789b0 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:55.582780 19028 tablet_service.cc:1467] Processing CreateTablet for tablet 29f27ed2c53d43d7a51ee2e45afaf9fc (DEFAULT_TABLE table=default.Table_b48d4141dcc04405a8797e7f7ccd3f53 [id=3c56915488654cc39eb7aade5982cfe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.583828 19034 tablet_service.cc:1467] Processing CreateTablet for tablet 8d20e8099b964fbc9e8e41447772d8d5 (DEFAULT_TABLE table=default.Table_6cdffa2e029846c3b95c8440c221426c [id=354a3a4b0b774fb9bc83623b2cb87f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.584106 19017 tablet_service.cc:1514] Processing DeleteTablet for tablet d83cfd85a52f438dabe49b6223d5090f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:54 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:55.591547 19034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d20e8099b964fbc9e8e41447772d8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.592233 19028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f27ed2c53d43d7a51ee2e45afaf9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.598989 19016 tablet_service.cc:1467] Processing CreateTablet for tablet 3d873a0faf4a4645a3969067ae0583dc (DEFAULT_TABLE table=default.Table_c9582d98645645f3aa80a65f7652cba8 [id=2d5d8f8a7ade443f91a7da34add25a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.607865 19015 tablet_service.cc:1467] Processing CreateTablet for tablet 218b87091d9546ba859f954a8f9e089e (DEFAULT_TABLE table=default.Table_c398ec562f594aca8c1e62cde38c04f9 [id=14bc02cbe30c486782d8fa4396855a9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.654069 19018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0bd71c6f4d4c90a4f3ea747af02165. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.600824 19030 tablet_service.cc:1514] Processing DeleteTablet for tablet 07f15096ea484b499fbdecec74eb6f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:54 UTC) from {username='slave'} at 127.0.0.1:36190
I20250111 02:57:55.663463 19030 tablet_service.cc:1467] Processing CreateTablet for tablet a2eae136623e4802b083d6526c6867c9 (DEFAULT_TABLE table=default.Table_97e52a18fbd44f189d46ba9b26cd9033 [id=566e89e7552f4ce08fba16fe299b9ccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.666918 20692 raft_consensus.cc:491] T f9092a3453f94abbbf4bdd66a57d42cc P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:55.667444 20692 raft_consensus.cc:513] T f9092a3453f94abbbf4bdd66a57d42cc P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:55.669338 20692 leader_election.cc:290] T f9092a3453f94abbbf4bdd66a57d42cc P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:55.680099 19016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d873a0faf4a4645a3969067ae0583dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.676430 19025 tablet_service.cc:1467] Processing CreateTablet for tablet 350f7c7f14004bb4bc3cb2338c7f172f (DEFAULT_TABLE table=default.Table_28d700d6f8444cfeab3dc52dd2bec17f [id=0e5eefce9dfa453eb277f75189def89e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.722270 20718 tablet_bootstrap.cc:492] T 0d4ba33836f04ed19d67b8bc161e57c4 P 54628fbcec2f4359aa9e26a734d4d29f: No bootstrap required, opened a new log
I20250111 02:57:55.722851 20718 ts_tablet_manager.cc:1397] T 0d4ba33836f04ed19d67b8bc161e57c4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent bootstrapping tablet: real 0.238s user 0.015s sys 0.010s
W20250111 02:57:55.733704 18617 connection.cc:537] client connection to 127.18.20.65:34535 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:57:55.734102 18617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
I20250111 02:57:55.735785 19015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 218b87091d9546ba859f954a8f9e089e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:55.738284 20718 raft_consensus.cc:357] T 0d4ba33836f04ed19d67b8bc161e57c4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:55.739048 20718 raft_consensus.cc:383] T 0d4ba33836f04ed19d67b8bc161e57c4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:55.739351 20718 raft_consensus.cc:738] T 0d4ba33836f04ed19d67b8bc161e57c4 P 54628fbcec2f4359aa9e26a734d4d29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54628fbcec2f4359aa9e26a734d4d29f, State: Initialized, Role: FOLLOWER
I20250111 02:57:55.739972 20718 consensus_queue.cc:260] T 0d4ba33836f04ed19d67b8bc161e57c4 P 54628fbcec2f4359aa9e26a734d4d29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:55.742396 20718 ts_tablet_manager.cc:1428] T 0d4ba33836f04ed19d67b8bc161e57c4 P 54628fbcec2f4359aa9e26a734d4d29f: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
W20250111 02:57:55.746721 19133 connection.cc:537] client connection to 127.18.20.65:34535 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:57:55.747220 19133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250111 02:57:55.758832 18513 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu with pid 18959
I20250111 02:57:55.765708 20736 tablet_replica.cc:331] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
W20250111 02:57:55.782066 19133 consensus_peers.cc:487] T 933828fd6fb041e9a4889f4ea8a3c5ff P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.65:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:55.782770 19133 consensus_peers.cc:487] T f778763e59a94969b3f62be87e2fd867 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.65:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:55.783262 19133 consensus_peers.cc:487] T d8ef3f45e5784a29b3b32cc0de6bac84 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.65:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:55.783712 19133 consensus_peers.cc:487] T a983a670c2104398ac1a159ac3a1da7c P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.65:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:55.784133 19133 leader_election.cc:336] T f9092a3453f94abbbf4bdd66a57d42cc P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Network error: Client connection negotiation failed: client connection to 127.18.20.65:34535: connect: Connection refused (error 111)
I20250111 02:57:55.785228 20736 raft_consensus.cc:2238] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:55.787797 20736 raft_consensus.cc:2267] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:55.821197 20722 tablet_bootstrap.cc:654] T dea4db154779483cb069118e3f3789b0 P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
W20250111 02:57:55.829823 18617 catalog_manager.cc:4615] TS 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): CreateTablet RPC failed for tablet 3d873a0faf4a4645a3969067ae0583dc: Network error: Client connection negotiation failed: client connection to 127.18.20.65:34535: connect: Connection refused (error 111)
I20250111 02:57:55.832263 20736 ts_tablet_manager.cc:1905] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:55.885035 20722 tablet_bootstrap.cc:492] T dea4db154779483cb069118e3f3789b0 P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:55.885650 20722 ts_tablet_manager.cc:1397] T dea4db154779483cb069118e3f3789b0 P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.307s user 0.015s sys 0.000s
I20250111 02:57:55.888399 20722 raft_consensus.cc:357] T dea4db154779483cb069118e3f3789b0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:55.890236 19133 consensus_peers.cc:487] T 40fdec982cf643c68ef90798f4f65ad4 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.65:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:55.898566 20722 raft_consensus.cc:383] T dea4db154779483cb069118e3f3789b0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:55.898954 20722 raft_consensus.cc:738] T dea4db154779483cb069118e3f3789b0 P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:55.899825 20722 consensus_queue.cc:260] T dea4db154779483cb069118e3f3789b0 P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
W20250111 02:57:55.911209 19133 consensus_peers.cc:487] T 36f71f8dd2894d40b3bfdd20c1453787 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.65:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:55.922456 20722 ts_tablet_manager.cc:1428] T dea4db154779483cb069118e3f3789b0 P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20250111 02:57:55.939811 20722 tablet_bootstrap.cc:492] T d2fc07e0474e4fc5a46f2350229ca31f P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:55.953289 20722 tablet_bootstrap.cc:654] T d2fc07e0474e4fc5a46f2350229ca31f P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:55.946516 20736 ts_tablet_manager.cc:1918] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:55.977100 20736 log.cc:1198] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/07f15096ea484b499fbdecec74eb6f66
I20250111 02:57:55.979697 20736 ts_tablet_manager.cc:1939] T 07f15096ea484b499fbdecec74eb6f66 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:55.987270 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet 07f15096ea484b499fbdecec74eb6f66 (table default.Table_7653ac6289d7467590ac03b43df1b8bf [id=043fd6ad4f5c4a2685af5bbecda90328]) successfully deleted
I20250111 02:57:55.990942 19172 tablet_service.cc:1467] Processing CreateTablet for tablet 7d6dcab379ab40cb8508adb9e909c816 (DEFAULT_TABLE table=default.Table_ccb78dfa1e94441ca883240c2d85ca5b [id=2ce7a215918c4f7f817373f5e6a37f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:55.992202 19172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d6dcab379ab40cb8508adb9e909c816. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:57:55.993654 19133 consensus_peers.cc:487] T 49616e4e7cc647509e72cde57b87d6fa P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.65:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:56.002027 19169 tablet_service.cc:1467] Processing CreateTablet for tablet 3150c8b1e34f434787be02fd6d83fc06 (DEFAULT_TABLE table=default.Table_5b39045192764dd8a67ddaab57d02a5f [id=8e3f1719d545435fa1c0d0f08b6b31ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:56.003249 19169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3150c8b1e34f434787be02fd6d83fc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:56.007881 19176 tablet_service.cc:1467] Processing CreateTablet for tablet b77f758653fd4b158b25624e1004386c (DEFAULT_TABLE table=default.Table_e2a51ae779294deb90786269e7a8f13d [id=746d96ba5f5f45f58e941251f2637457]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:56.008944 19176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b77f758653fd4b158b25624e1004386c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:56.024919 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet fa86b1e6eca7403c9a9c52592b8fb636 (table default.Table_4b9e85e01e1843ce831205177cbb2e0f [id=0a118da8c5d0408980cbd0b33e0c2ed0]). Expected version 1 got 0
I20250111 02:57:56.029562 19162 tablet_service.cc:1467] Processing CreateTablet for tablet e93c0b5638184468a9090e7e316461e3 (DEFAULT_TABLE table=default.Table_5d8fd938fc9749e8a833fe51b755d938 [id=1893c90ebfc64ed297da93604d14d777]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:56.030696 19162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e93c0b5638184468a9090e7e316461e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:56.033149 20722 tablet_bootstrap.cc:492] T d2fc07e0474e4fc5a46f2350229ca31f P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:56.033596 20722 ts_tablet_manager.cc:1397] T d2fc07e0474e4fc5a46f2350229ca31f P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.094s user 0.037s sys 0.038s
I20250111 02:57:56.035471 20736 tablet_replica.cc:331] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:56.036387 20736 raft_consensus.cc:2238] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:56.039780 20736 raft_consensus.cc:2267] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:56.041250 19165 tablet_service.cc:1467] Processing CreateTablet for tablet 5e03d4ebb40746e484b0fb4f4b2dd35d (DEFAULT_TABLE table=default.Table_02b5991089a34b0b8ee6039ccc092c93 [id=977f59b134154bc396bf42bc1e44e717]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:56.042380 19165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e03d4ebb40746e484b0fb4f4b2dd35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:56.042749 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet a983a670c2104398ac1a159ac3a1da7c (table default.Table_745da7eb612249d8b494bee5c97df63a [id=a37b721314134c86b39d1f80b7359f19]). Expected version 5 got 4
I20250111 02:57:56.043002 20722 raft_consensus.cc:357] T d2fc07e0474e4fc5a46f2350229ca31f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:56.045500 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 855dac47872d4c5fbf8f5e38b588c7fe (table default.Table_49e4dd58d29f4cb6a0ed4f85138713b8 [id=f1373922978a4598b888e1863525942e]). Expected version 1 got 0
I20250111 02:57:56.046142 19175 tablet_service.cc:1467] Processing CreateTablet for tablet 5766b6e8bc2d4d9683abd39fba8d2089 (DEFAULT_TABLE table=default.Table_6d5220f0af774036a2c0a13c8eb24728 [id=01db2991d64744a5bffc9a0f859d5609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:57:56.046442 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 6a03880a8b0e46b7858ed1c2c2dddcaa (table default.Table_0158103117c842e0ae88e8f041658357 [id=5d8f4643061740aa96b23a6d4cba8038]). Expected version 4 got 0
I20250111 02:57:56.047156 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 3514d795a0f24adcaa2accc0cf05b142 (table default.Table_4c9aebb06d554107a5448d968fdb0436 [id=3fab00618e6548de85b5d20adee4e9ac]). Expected version 1 got 0
I20250111 02:57:56.047312 19175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5766b6e8bc2d4d9683abd39fba8d2089. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:57:56.047926 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet b4273afc264843029ed0baf1fc5c8a3e (table default.Table_4e3a79a5d2184687a8c51fd2702c5fd0 [id=73c59c0d9864497fbc9f19e265d8283f]). Expected version 1 got 0
I20250111 02:57:56.048535 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet e46894710a72407993b36801f3b90eb0 (table default.Table_04353d7ad63649d4bf1946d138f8b553 [id=d2dff624ad604b4c8746d5ba6ed87de3]). Expected version 2 got 0
I20250111 02:57:56.049088 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 220da826a0f140d59193e8ad72096b60 (table default.Table_5e7c139097914dd7b7c8e55058d5290d [id=9143d1d3b8754ac78d615a1cc4601d86]). Expected version 2 got 0
I20250111 02:57:56.049620 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet ffd063f8889b4bcdb9656466f0b144f3 (table default.Table_53d44be8eaeb4e59a8e3454aac56daca [id=8de61815471842e593a7d29efffc443c]). Expected version 1 got 0
I20250111 02:57:56.050204 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet ea76b354a75041e888cf862250e4947b (table default.Table_b44f1ebbaf7045e2bcfefcee1c91c0d9 [id=ec5cf5f9b6234008914c0ed664260dcb]). Expected version 1 got 0
I20250111 02:57:56.050699 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet e4bd42ad66064bcaa043746081f1c759 (table default.Table_ef8a731a05e841b6bd31bd3df1a8e375 [id=a0260905798243e9a870e755912fb0b7]). Expected version 2 got 0
I20250111 02:57:56.053618 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet 845da3ee0863430ea3fe9de8c84f2c17 (table default.Table_83e6031ed7fe4cbe88502007987425a1 [id=b091f6cd63104693abbdd739bca159c3]). Expected version 2 got 0
I20250111 02:57:56.055284 18630 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: default.Table_019bed496592413585d882a5d46666cb [id=94bfb73969874d449273803ecdb4a605] alter complete (version 2)
I20250111 02:57:56.055825 20736 ts_tablet_manager.cc:1905] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:56.043787 20722 raft_consensus.cc:383] T d2fc07e0474e4fc5a46f2350229ca31f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:56.063256 20722 raft_consensus.cc:738] T d2fc07e0474e4fc5a46f2350229ca31f P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:56.064184 20722 consensus_queue.cc:260] T d2fc07e0474e4fc5a46f2350229ca31f P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:56.066798 20722 ts_tablet_manager.cc:1428] T d2fc07e0474e4fc5a46f2350229ca31f P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20250111 02:57:56.087126 20722 tablet_bootstrap.cc:492] T 788c34eff1884d979add7e09615c02ae P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
W20250111 02:57:56.094802 19133 consensus_peers.cc:487] T e46894710a72407993b36801f3b90eb0 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.65:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:56.106226 20722 tablet_bootstrap.cc:654] T 788c34eff1884d979add7e09615c02ae P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:56.134851 18630 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8784b9603f8544b79373158aace063af: default.Table_842f087ec75f45f2a6ba5652f3a0bde1 [id=cf9df41fb95c4e9c89cf0f52217843a5] alter complete (version 1)
I20250111 02:57:56.153638 20736 ts_tablet_manager.cc:1918] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:56.154006 20736 log.cc:1198] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/d83cfd85a52f438dabe49b6223d5090f
I20250111 02:57:56.157230 20722 tablet_bootstrap.cc:492] T 788c34eff1884d979add7e09615c02ae P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:56.157797 20722 ts_tablet_manager.cc:1397] T 788c34eff1884d979add7e09615c02ae P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.007s
I20250111 02:57:56.160656 20722 raft_consensus.cc:357] T 788c34eff1884d979add7e09615c02ae P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:56.169631 20722 raft_consensus.cc:383] T 788c34eff1884d979add7e09615c02ae P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250111 02:57:56.169059 19133 consensus_peers.cc:487] T 9d6de202f4e547a0b8b90cfa09ad319a P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.65:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:57:56.162755 20736 ts_tablet_manager.cc:1939] T d83cfd85a52f438dabe49b6223d5090f P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:56.173228 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet d83cfd85a52f438dabe49b6223d5090f (table default.Table_799d0b8328de40a7bacb58ea433fec72 [id=e9efdc6c94574ddca092dd6b9f3b8419]) successfully deleted
I20250111 02:57:56.173916 20722 raft_consensus.cc:738] T 788c34eff1884d979add7e09615c02ae P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:56.179621 20722 consensus_queue.cc:260] T 788c34eff1884d979add7e09615c02ae P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:56.190495 20722 ts_tablet_manager.cc:1428] T 788c34eff1884d979add7e09615c02ae P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20250111 02:57:56.203334 19162 tablet_service.cc:1514] Processing DeleteTablet for tablet b26e5db463e94d919640a07a2f8574a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:22 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:56.204630 20722 tablet_bootstrap.cc:492] T 12065effb54b4d4a94349641109b578c P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:56.204953 20732 raft_consensus.cc:491] T 788c34eff1884d979add7e09615c02ae P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:57:56.205425 20732 raft_consensus.cc:513] T 788c34eff1884d979add7e09615c02ae P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:56.206132 20736 tablet_replica.cc:331] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:56.207152 20736 raft_consensus.cc:2238] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:56.207584 20732 leader_election.cc:290] T 788c34eff1884d979add7e09615c02ae P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535), 54628fbcec2f4359aa9e26a734d4d29f (127.18.20.66:44623)
I20250111 02:57:56.207747 20736 raft_consensus.cc:2267] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:57:56.210525 20736 ts_tablet_manager.cc:1905] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:57:56.214565 19162 tablet_service.cc:1514] Processing DeleteTablet for tablet 3052e9707de14e1f9e38236d56908ed3 with delete_type TABLET_DATA_DELETED (replaced by tablet f31ad2e1895d4a26b8e9f7cd7366ef81) from {username='slave'} at 127.0.0.1:32868
W20250111 02:57:56.214838 19133 leader_election.cc:336] T 788c34eff1884d979add7e09615c02ae P d0cedab82b154d62b78809edc6f28bfa [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Network error: Client connection negotiation failed: client connection to 127.18.20.65:34535: connect: Connection refused (error 111)
I20250111 02:57:56.225713 19175 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ef86e1fd412484db83e7d39c9059a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:23 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:56.235786 20722 tablet_bootstrap.cc:654] T 12065effb54b4d4a94349641109b578c P d0cedab82b154d62b78809edc6f28bfa: Neither blocks nor log segments found. Creating new log.
I20250111 02:57:56.236789 19162 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e06e867266948df8eb35a85c58f578f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:20 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:56.238834 19162 tablet_service.cc:1514] Processing DeleteTablet for tablet 42230a0bc94249fda91cf79fba48f024 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:24 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:56.240098 19162 tablet_service.cc:1514] Processing DeleteTablet for tablet 2873325cece64db6a673d4fe08eb7409 with delete_type TABLET_DATA_DELETED (replaced by tablet 3d873a0faf4a4645a3969067ae0583dc) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:56.241259 19162 tablet_service.cc:1514] Processing DeleteTablet for tablet 07f15096ea484b499fbdecec74eb6f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:54 UTC) from {username='slave'} at 127.0.0.1:32868
W20250111 02:57:56.242329 18630 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.66:37049 (request call id 141) took 2167 ms (client timeout 1000 ms). Trace:
I20250111 02:57:56.252714 19162 tablet_service.cc:1514] Processing DeleteTablet for tablet 076aafe3958948079e42501c6bd86153 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:26 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:56.254292 19175 tablet_service.cc:1514] Processing DeleteTablet for tablet 061e6eb5775443679e9cbd6b11f66fe3 with delete_type TABLET_DATA_DELETED (replaced by tablet 218b87091d9546ba859f954a8f9e089e) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:56.255470 19175 tablet_service.cc:1514] Processing DeleteTablet for tablet 48dd0bccf6854d37a7ef95b5ab06308e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:24 UTC) from {username='slave'} at 127.0.0.1:32868
I20250111 02:57:56.256218 19162 tablet_service.cc:1514] Processing DeleteTablet for tablet d83cfd85a52f438dabe49b6223d5090f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:57:54 UTC) from {username='slave'} at 127.0.0.1:32868
W20250111 02:57:56.257602 18628 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.20.67:53171 (request call id 134) took 848 ms (client timeout 1000 ms). Trace:
W20250111 02:57:56.257948 18628 rpcz_store.cc:269] 0111 02:57:55.409634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:55.409784 (+ 150us) service_pool.cc:225] Handling call
0111 02:57:56.257541 (+847757us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23444,"reported_tablets":63}
W20250111 02:57:56.264196 19133 consensus_peers.cc:487] T 64abb30da7fc452cad774bbe59b498f4 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.65:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:56.264683 19133 consensus_peers.cc:487] T 9192160e931445f1acd207604192f759 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.65:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:56.282641 19133 consensus_peers.cc:487] T 076aafe3958948079e42501c6bd86153 P d0cedab82b154d62b78809edc6f28bfa -> Peer 8221604733fe4fc0a1fc32de8d6af71c (127.18.20.65:34535): Couldn't send request to peer 8221604733fe4fc0a1fc32de8d6af71c. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.65:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:56.242664 18630 rpcz_store.cc:269] 0111 02:57:54.074772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:57:54.190987 (+116215us) service_pool.cc:225] Handling call
0111 02:57:56.134457 (+1943470us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:56.134486 (+ 29us) write_op.cc:454] Released schema lock
0111 02:57:56.242254 (+107768us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:55.588937 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:55.605660 (+ 16723us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:55.605691 (+ 31us) write_op.cc:435] Acquired schema lock
0111 02:57:55.605709 (+ 18us) tablet.cc:659] Decoding operations
0111 02:57:55.621198 (+ 15489us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x7b1800083d40. stack: 00007fd421031853 00007fd421031750 00007fd420ba6584 00000000004fcd24 00000000004fccb9 00007fd429c76931 00007fd429c83831 00007fd427dd208c 00007fd4210464ec 000000000058ec29 00007fd42106335b 00007fd42106175f 00007fd421060bf5 00007fd428412f9c 00007fd4283d3a68 00007fd429d843a1
0111 02:57:55.634971 (+ 13773us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:55.635051 (+ 80us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:55.635108 (+ 57us) tablet.cc:682] Acquiring locks for 40 operations
0111 02:57:55.692115 (+ 57007us) tablet.cc:698] Row locks acquired
0111 02:57:55.692132 (+ 17us) write_op.cc:260] PREPARE: finished
0111 02:57:55.692277 (+ 145us) write_op.cc:270] Start()
0111 02:57:55.692465 (+ 188us) write_op.cc:276] Timestamp: P: 1736564275692181 usec, L: 0
0111 02:57:55.692484 (+ 19us) op_driver.cc:348] REPLICATION: starting
0111 02:57:55.693183 (+ 699us) log.cc:844] Serialized 15234 byte log entry
0111 02:57:55.715599 (+ 22416us) op_driver.cc:464] REPLICATION: finished
0111 02:57:55.810904 (+ 95305us) write_op.cc:301] APPLY: starting
0111 02:57:55.810987 (+ 83us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:55.811997 (+ 1010us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:55.812027 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:55.847868 (+ 35841us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:55.848518 (+ 650us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=40
0111 02:57:55.848588 (+ 70us) write_op.cc:312] APPLY: finished
0111 02:57:55.849444 (+ 856us) log.cc:844] Serialized 260 byte log entry
0111 02:57:55.849644 (+ 200us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:55.849916 (+ 272us) write_op.cc:454] Released schema lock
0111 02:57:55.849994 (+ 78us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:55.850259 (+ 265us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:55.850284 (+ 25us) write_op.cc:454] Released schema lock
Related trace 'op':
0111 02:57:55.872011 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:55.872462 (+ 451us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:55.872490 (+ 28us) write_op.cc:435] Acquired schema lock
0111 02:57:55.872501 (+ 11us) tablet.cc:659] Decoding operations
0111 02:57:55.872852 (+ 351us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:55.872900 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:55.872928 (+ 28us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:55.873087 (+ 159us) tablet.cc:698] Row locks acquired
0111 02:57:55.873100 (+ 13us) write_op.cc:260] PREPARE: finished
0111 02:57:55.873239 (+ 139us) write_op.cc:270] Start()
0111 02:57:55.873331 (+ 92us) write_op.cc:276] Timestamp: P: 1736564275873141 usec, L: 0
0111 02:57:55.873346 (+ 15us) op_driver.cc:348] REPLICATION: starting
0111 02:57:55.873895 (+ 549us) log.cc:844] Serialized 356 byte log entry
0111 02:57:55.882807 (+ 8912us) op_driver.cc:464] REPLICATION: finished
0111 02:57:55.891822 (+ 9015us) write_op.cc:301] APPLY: starting
0111 02:57:55.891879 (+ 57us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:55.891970 (+ 91us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:55.892057 (+ 87us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:55.892239 (+ 182us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:55.892365 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:55.892415 (+ 50us) write_op.cc:312] APPLY: finished
0111 02:57:55.894897 (+ 2482us) log.cc:844] Serialized 23 byte log entry
0111 02:57:55.895617 (+ 720us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:55.895687 (+ 70us) write_op.cc:454] Released schema lock
0111 02:57:55.895721 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0111 02:57:55.895959 (+ 238us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:55.895986 (+ 27us) write_op.cc:454] Released schema lock
Related trace 'op':
0111 02:57:56.065788 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0111 02:57:56.066239 (+ 451us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:56.066266 (+ 27us) write_op.cc:435] Acquired schema lock
0111 02:57:56.066283 (+ 17us) tablet.cc:659] Decoding operations
0111 02:57:56.066585 (+ 302us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:56.066628 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:56.066653 (+ 25us) tablet.cc:682] Acquiring locks for 1 operations
0111 02:57:56.066848 (+ 195us) tablet.cc:698] Row locks acquired
0111 02:57:56.066867 (+ 19us) write_op.cc:260] PREPARE: finished
0111 02:57:56.066994 (+ 127us) write_op.cc:270] Start()
0111 02:57:56.067124 (+ 130us) write_op.cc:276] Timestamp: P: 1736564276066893 usec, L: 0
0111 02:57:56.067144 (+ 20us) op_driver.cc:348] REPLICATION: starting
0111 02:57:56.067718 (+ 574us) log.cc:844] Serialized 356 byte log entry
0111 02:57:56.090086 (+ 22368us) op_driver.cc:464] REPLICATION: finished
0111 02:57:56.090415 (+ 329us) write_op.cc:301] APPLY: starting
0111 02:57:56.090467 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:56.090576 (+ 109us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:56.090682 (+ 106us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:56.090868 (+ 186us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:56.090996 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0111 02:57:56.091040 (+ 44us) write_op.cc:312] APPLY: finished
0111 02:57:56.091261 (+ 221us) log.cc:844] Serialized 23 byte log entry
0111 02:57:56.091406 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:56.091473 (+ 67us) write_op.cc:454] Released schema lock
0111 02:57:56.091501 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":3929,"reported_tablets":68,"child_traces":[["op",{"apply.queue_time_us":95065,"apply.run_cpu_time_us":7900,"apply.run_wall_time_us":39346,"num_ops":40,"prepare.queue_time_us":171567,"prepare.run_cpu_time_us":10358,"prepare.run_wall_time_us":118580,"raft.queue_time_us":1413,"raft.run_cpu_time_us":1884,"raft.run_wall_time_us":1884,"replication_time_us":23008,"spinlock_wait_cycles":75011840,"thread_start_us":288029,"threads_started":3}],["op",{"apply.queue_time_us":8822,"apply.run_cpu_time_us":1270,"apply.run_wall_time_us":4114,"num_ops":1,"prepare.queue_time_us":5009,"prepare.run_cpu_time_us":2254,"prepare.run_wall_time_us":2254,"raft.queue_time_us":2894,"raft.run_cpu_time_us":1542,"raft.run_wall_time_us":1542,"replication_time_us":9353}],["op",{"apply.queue_time_us":159,"apply.run_cpu_time_us":1267,"apply.run_wall_time_us":1267,"num_ops":1,"prepare.queue_time_us":105,"prepare.run_cpu_time_us":2290,"prepare.run_wall_time_us":2291,"raft.queue_time_us":3679,"raft.run_cpu_time_us":1272,"raft.run_wall_time_us":15206,"replication_time_us":22839}]]}
I20250111 02:57:56.301215 20722 tablet_bootstrap.cc:492] T 12065effb54b4d4a94349641109b578c P d0cedab82b154d62b78809edc6f28bfa: No bootstrap required, opened a new log
I20250111 02:57:56.303460 20722 ts_tablet_manager.cc:1397] T 12065effb54b4d4a94349641109b578c P d0cedab82b154d62b78809edc6f28bfa: Time spent bootstrapping tablet: real 0.099s user 0.015s sys 0.029s
I20250111 02:57:56.306524 20722 raft_consensus.cc:357] T 12065effb54b4d4a94349641109b578c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:56.307569 20722 raft_consensus.cc:383] T 12065effb54b4d4a94349641109b578c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:57:56.307966 20722 raft_consensus.cc:738] T 12065effb54b4d4a94349641109b578c P d0cedab82b154d62b78809edc6f28bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cedab82b154d62b78809edc6f28bfa, State: Initialized, Role: FOLLOWER
I20250111 02:57:56.301951 20736 ts_tablet_manager.cc:1918] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:57:56.309396 20736 log.cc:1198] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/b26e5db463e94d919640a07a2f8574a8
I20250111 02:57:56.309123 20722 consensus_queue.cc:260] T 12065effb54b4d4a94349641109b578c P d0cedab82b154d62b78809edc6f28bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221604733fe4fc0a1fc32de8d6af71c" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34535 } } peers { permanent_uuid: "54628fbcec2f4359aa9e26a734d4d29f" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 44623 } } peers { permanent_uuid: "d0cedab82b154d62b78809edc6f28bfa" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 35591 } }
I20250111 02:57:56.312256 20722 ts_tablet_manager.cc:1428] T 12065effb54b4d4a94349641109b578c P d0cedab82b154d62b78809edc6f28bfa: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20250111 02:57:56.312566 20736 ts_tablet_manager.cc:1939] T b26e5db463e94d919640a07a2f8574a8 P d0cedab82b154d62b78809edc6f28bfa: Deleting consensus metadata
I20250111 02:57:56.314143 18628 catalog_manager.cc:5618] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591) does not have the latest schema for tablet dea4db154779483cb069118e3f3789b0 (table default.Table_8eae98e1e49c4f6c9e52644256812eaa [id=7dfae414add74b6491f2f7595a551342]). Expected version 1 got 0
W20250111 02:57:56.318557 18628 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet dea4db154779483cb069118e3f3789b0 (table default.Table_8eae98e1e49c4f6c9e52644256812eaa [id=7dfae414add74b6491f2f7595a551342], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet dea4db154779483cb069118e3f3789b0 consensus state has no leader
I20250111 02:57:56.324033 20722 tablet_bootstrap.cc:492] T c8f0c42e06d4428284fa006fe0062cf0 P d0cedab82b154d62b78809edc6f28bfa: Bootstrap starting.
I20250111 02:57:56.325609 20736 tablet_replica.cc:331] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa: stopping tablet replica
I20250111 02:57:56.326562 20736 raft_consensus.cc:2238] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:57:56.327697 18614 catalog_manager.cc:4872] TS d0cedab82b154d62b78809edc6f28bfa (127.18.20.67:35591): tablet b26e5db463e94d919640a07a2f8574a8 (table default.Table_75e5f05a97b14bf883909660868761fd [id=33afe1cfccda4b79b1aaf539f56f27f9]) successfully deleted
I20250111 02:57:56.328408 20736 raft_consensus.cc:2267] T 3052e9707de14e1f9e38236d56908ed3 P d0cedab82b154d62b78809edc6f28bfa [term 1 FOLLOWER]: Raft consensus is shut down!
W20250111 02:57:56.336238 18615 connection.cc:537] client connection to 127.18.20.66:44623 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:57:56.343600 18614 connection.cc:537] server connection from 127.18.20.66:37049 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250111 02:57:56.347968 18513 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu with pid 19110
W20250111 02:57:56.371654 18614 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet dea4db154779483cb069118e3f3789b0 (table default.Table_8eae98e1e49c4f6c9e52644256812eaa [id=7dfae414add74b6491f2f7595a551342], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet dea4db154779483cb069118e3f3789b0 consensus state has no leader
W20250111 02:57:56.446408 19594 connection.cc:537] server connection from 127.18.20.67:41951 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250111 02:57:56.448400 18513 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu with pid 19309
I20250111 02:57:56.499676 18513 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu with pid 18592
I20250111 02:57:56.560444 18513 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu with pid 18667
2025-01-11T02:57:56Z chronyd exiting
I20250111 02:57:56.615947 18513 master-stress-test.cc:585] Tables created: 127
I20250111 02:57:56.616155 18513 master-stress-test.cc:586] Tables altered: 55
I20250111 02:57:56.616328 18513 master-stress-test.cc:587] Tables deleted: 45
I20250111 02:57:56.616497 18513 master-stress-test.cc:588] Tablets replaced: 22
I20250111 02:57:56.616657 18513 master-stress-test.cc:589] Masters restarted: 1
[ OK ] CatalogManagerConfigurations/MasterStressTest.Test/0 (58325 ms)
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/3
I20250111 02:57:56.708473 18513 test_util.cc:274] Using random seed: 1624753247
2025-01-11T02:57:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-11T02:57:56Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskY_5DOU/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskY_5DOU/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/security.properties
2025-01-11 02:58:03: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskY_5DOU/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskY_5DOU/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250111 02:58:12.716475 18513 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
/tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.126:34627
--webserver_interface=127.18.20.126
--webserver_port=0
--builtin_ntp_servers=127.18.20.84:35279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--hive_metastore_uris=thrift://127.0.0.1:41043
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250111 02:58:13.039912 21002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:58:13.040501 21002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:58:13.040932 21002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:58:13.072940 21002 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250111 02:58:13.073202 21002 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250111 02:58:13.073460 21002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 02:58:13.073835 21002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:58:13.074177 21002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 02:58:13.074442 21002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 02:58:13.113031 21002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.20.84:35279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:41043
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.126:34627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.20.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4120
TSAN enabled
I20250111 02:58:13.114398 21002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:58:13.116150 21002 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:58:13.134325 21008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:58:13.134770 21009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:58:13.136210 21011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:13.137421 21002 server_base.cc:1034] running on GCE node
I20250111 02:58:14.324363 21002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:58:14.327277 21002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:58:14.328730 21002 hybrid_clock.cc:648] HybridClock initialized: now 1736564294328676 us; error 57 us; skew 500 ppm
I20250111 02:58:14.329668 21002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:58:14.336680 21002 webserver.cc:458] Webserver started at http://127.18.20.126:35247/ using document root <none> and password file <none>
I20250111 02:58:14.337740 21002 fs_manager.cc:362] Metadata directory not provided
I20250111 02:58:14.337987 21002 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:58:14.338482 21002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:58:14.344918 21002 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/data/instance:
uuid: "3476b1c08aec44598f1c203c2cbcf016"
format_stamp: "Formatted at 2025-01-11 02:58:14 on dist-test-slave-09x0"
I20250111 02:58:14.346925 21002 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/wal/instance:
uuid: "3476b1c08aec44598f1c203c2cbcf016"
format_stamp: "Formatted at 2025-01-11 02:58:14 on dist-test-slave-09x0"
I20250111 02:58:14.358726 21002 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.010s sys 0.000s
I20250111 02:58:14.367781 21018 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:58:14.369604 21002 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20250111 02:58:14.370030 21002 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/data,/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/wal
uuid: "3476b1c08aec44598f1c203c2cbcf016"
format_stamp: "Formatted at 2025-01-11 02:58:14 on dist-test-slave-09x0"
I20250111 02:58:14.370546 21002 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:58:14.430562 21002 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:58:14.433013 21002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:58:14.433707 21002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:58:14.510061 21002 rpc_server.cc:307] RPC server started. Bound to: 127.18.20.126:34627
I20250111 02:58:14.510133 21069 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.20.126:34627 every 8 connection(s)
I20250111 02:58:14.513325 21002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/data/info.pb
I20250111 02:58:14.518786 21070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:14.523716 18513 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu as pid 21002
I20250111 02:58:14.524132 18513 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/wal/instance
I20250111 02:58:14.526707 18513 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
/tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.125:35029
--webserver_interface=127.18.20.125
--webserver_port=0
--builtin_ntp_servers=127.18.20.84:35279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--hive_metastore_uris=thrift://127.0.0.1:41043
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250111 02:58:14.530973 21070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:58:14.557986 21070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } has no permanent_uuid. Determining permanent_uuid...
W20250111 02:58:14.560712 21019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:58:14.565572 21070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 02:58:14.615171 21070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } attempt: 1
W20250111 02:58:14.620265 21070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 02:58:14.688712 21070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } attempt: 2
W20250111 02:58:14.693640 21070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 02:58:14.785079 21070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } attempt: 3
W20250111 02:58:14.789445 21070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:58:14.853518 21071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:58:14.854153 21071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:58:14.854636 21071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:58:14.886639 21071 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250111 02:58:14.886947 21071 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250111 02:58:14.887249 21071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 02:58:14.887508 21071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:58:14.887754 21071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 02:58:14.887987 21071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 02:58:14.925179 21071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.20.84:35279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41043
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.125:35029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.20.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4120
TSAN enabled
I20250111 02:58:14.926651 21071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:58:14.928390 21071 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 02:58:14.932873 21070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } attempt: 4
W20250111 02:58:14.937983 21070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:58:14.945915 21081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:15.237604 21070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } attempt: 5
W20250111 02:58:15.244037 21070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 02:58:15.791486 21070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } attempt: 6
W20250111 02:58:15.800274 21070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:58:14.946767 21080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:14.949018 21071 server_base.cc:1034] running on GCE node
W20250111 02:58:14.948396 21083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:16.137658 21071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:58:16.141053 21071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:58:16.142581 21071 hybrid_clock.cc:648] HybridClock initialized: now 1736564296142529 us; error 66 us; skew 500 ppm
I20250111 02:58:16.143517 21071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:58:16.151345 21071 webserver.cc:458] Webserver started at http://127.18.20.125:38137/ using document root <none> and password file <none>
I20250111 02:58:16.152366 21071 fs_manager.cc:362] Metadata directory not provided
I20250111 02:58:16.152593 21071 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:58:16.153069 21071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:58:16.158339 21071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/data/instance:
uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21"
format_stamp: "Formatted at 2025-01-11 02:58:16 on dist-test-slave-09x0"
I20250111 02:58:16.159952 21071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/wal/instance:
uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21"
format_stamp: "Formatted at 2025-01-11 02:58:16 on dist-test-slave-09x0"
I20250111 02:58:16.168037 21071 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250111 02:58:16.175117 21092 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:58:16.176452 21071 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250111 02:58:16.176826 21071 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/data,/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/wal
uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21"
format_stamp: "Formatted at 2025-01-11 02:58:16 on dist-test-slave-09x0"
I20250111 02:58:16.177181 21071 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:58:16.229355 21071 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:58:16.230882 21071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:58:16.231389 21071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:58:16.306553 21071 rpc_server.cc:307] RPC server started. Bound to: 127.18.20.125:35029
I20250111 02:58:16.306663 21143 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.20.125:35029 every 8 connection(s)
I20250111 02:58:16.309535 21071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/data/info.pb
I20250111 02:58:16.314649 18513 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu as pid 21071
I20250111 02:58:16.315083 18513 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/wal/instance
I20250111 02:58:16.315500 21144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:16.317622 18513 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
/tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.124:36707
--webserver_interface=127.18.20.124
--webserver_port=0
--builtin_ntp_servers=127.18.20.84:35279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--hive_metastore_uris=thrift://127.0.0.1:41043
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250111 02:58:16.329552 21144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:58:16.359777 21144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:58:16.374584 21144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } has no permanent_uuid. Determining permanent_uuid...
W20250111 02:58:16.377343 21095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 02:58:16.382387 21144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:36707: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:16.431957 21144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } attempt: 1
W20250111 02:58:16.437393 21144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:36707: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:16.505957 21144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } attempt: 2
W20250111 02:58:16.511044 21144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:36707: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:16.602624 21144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } attempt: 3
W20250111 02:58:16.607407 21144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:36707: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 02:58:16.656268 21145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:58:16.657001 21145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:58:16.657500 21145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:58:16.689931 21145 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250111 02:58:16.690199 21145 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250111 02:58:16.690487 21145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 02:58:16.690732 21145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:58:16.690970 21145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 02:58:16.691262 21145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 02:58:16.728032 21145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.20.84:35279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41043
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.124:36707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.20.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4120
TSAN enabled
I20250111 02:58:16.729507 21145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:58:16.731204 21145 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:58:16.748499 21158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:16.750937 21144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } attempt: 4
W20250111 02:58:16.755182 21144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:36707: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:16.860700 21070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } attempt: 7
I20250111 02:58:16.873183 21070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } has no permanent_uuid. Determining permanent_uuid...
W20250111 02:58:16.880105 21070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:36707: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:16.938510 21070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } attempt: 1
W20250111 02:58:16.943495 21070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:36707: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:17.024945 21070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } attempt: 2
W20250111 02:58:17.030048 21070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:36707: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:17.054740 21144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } attempt: 5
W20250111 02:58:17.059820 21144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:36707: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:17.115556 21070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } attempt: 3
W20250111 02:58:17.120829 21070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:36707: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:17.261293 21070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } attempt: 4
W20250111 02:58:17.266351 21070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:36707: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:17.549814 21070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } attempt: 5
W20250111 02:58:17.554404 21070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:36707: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:17.607239 21144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } attempt: 6
W20250111 02:58:17.616328 21144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:36707: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:16.752032 21145 server_base.cc:1034] running on GCE node
W20250111 02:58:16.749229 21156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:58:16.750694 21155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:17.959463 21145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:58:17.962357 21145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:58:17.963773 21145 hybrid_clock.cc:648] HybridClock initialized: now 1736564297963739 us; error 37 us; skew 500 ppm
I20250111 02:58:17.964668 21145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:58:17.975934 21145 webserver.cc:458] Webserver started at http://127.18.20.124:41133/ using document root <none> and password file <none>
I20250111 02:58:17.977051 21145 fs_manager.cc:362] Metadata directory not provided
I20250111 02:58:17.977304 21145 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:58:17.977815 21145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:58:17.983175 21145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/data/instance:
uuid: "7ebb149f387f469fbb3731becb98426f"
format_stamp: "Formatted at 2025-01-11 02:58:17 on dist-test-slave-09x0"
I20250111 02:58:17.984481 21145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/wal/instance:
uuid: "7ebb149f387f469fbb3731becb98426f"
format_stamp: "Formatted at 2025-01-11 02:58:17 on dist-test-slave-09x0"
I20250111 02:58:17.992501 21145 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250111 02:58:17.998827 21169 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:58:18.000031 21145 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250111 02:58:18.000401 21145 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/data,/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/wal
uuid: "7ebb149f387f469fbb3731becb98426f"
format_stamp: "Formatted at 2025-01-11 02:58:17 on dist-test-slave-09x0"
I20250111 02:58:18.000778 21145 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:58:18.060746 21145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:58:18.062472 21145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:58:18.062978 21145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:58:18.106873 21070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } attempt: 6
W20250111 02:58:18.116753 21070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.20.124:36707: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:18.142233 21145 rpc_server.cc:307] RPC server started. Bound to: 127.18.20.124:36707
I20250111 02:58:18.142315 21222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.20.124:36707 every 8 connection(s)
I20250111 02:58:18.145444 21145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/data/info.pb
I20250111 02:58:18.146804 18513 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu as pid 21145
I20250111 02:58:18.147341 18513 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-2/wal/instance
I20250111 02:58:18.152318 21223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:18.169747 21223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:58:18.196630 21223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:58:18.207577 21223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:58:18.224306 21223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: Bootstrap starting.
I20250111 02:58:18.231292 21223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:18.233184 21223 log.cc:826] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: Log is configured to *not* fsync() on all Append() calls
I20250111 02:58:18.238947 21223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: No bootstrap required, opened a new log
I20250111 02:58:18.259857 21223 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:18.260875 21223 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:18.261242 21223 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ebb149f387f469fbb3731becb98426f, State: Initialized, Role: FOLLOWER
I20250111 02:58:18.262197 21223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:18.264768 21230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:18.265497 21230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:18.266290 21223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: configured and running, proceeding with master startup.
W20250111 02:58:18.305285 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.305828 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.318965 21175 tablet.cc:2367] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250111 02:58:18.320973 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.321480 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:58:18.296973 21145 master_runner.cc:186] Error getting master registration for 127.18.20.126:34627: OK, instance_id { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" instance_seqno: 1736564294489156 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250111 02:58:18.335453 21145 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 02:58:18.336709 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.337159 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.362803 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.363281 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.378463 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.378932 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.393657 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.394122 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.410931 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.411343 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:58:18.418640 21230 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:18.419335 21230 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:18.423571 21230 leader_election.cc:290] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029)
W20250111 02:58:18.426314 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.426728 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.441481 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.441869 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.456218 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.456498 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.470007 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.470305 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.484938 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.485311 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.500015 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.500406 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.515640 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.516078 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.535462 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.535835 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.550515 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.550938 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.565814 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.566186 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.581082 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.581473 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.596205 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.596603 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.611472 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.611871 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.626729 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.627141 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.642109 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.642496 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.657323 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.657713 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:58:18.672420 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.672858 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:58:18.677105 21144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } attempt: 7
W20250111 02:58:18.687631 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.688732 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:58:18.701805 21144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: Bootstrap starting.
W20250111 02:58:18.703805 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.704130 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:58:18.711712 21144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:18.714447 21144 log.cc:826] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: Log is configured to *not* fsync() on all Append() calls
W20250111 02:58:18.719023 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.719775 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:58:18.721314 21144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: No bootstrap required, opened a new log
W20250111 02:58:18.738628 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.739362 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:58:18.755473 21144 raft_consensus.cc:357] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:18.756435 21144 raft_consensus.cc:383] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:18.756759 21144 raft_consensus.cc:738] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3a224ebe0341df84e1d1d8fdfb0f21, State: Initialized, Role: FOLLOWER
I20250111 02:58:18.757824 21144 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
W20250111 02:58:18.759111 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.759721 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:58:18.761277 21249 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:18.762197 21249 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:18.763551 21144 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:58:18.763685 21119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" is_pre_election: true
I20250111 02:58:18.764737 21119 raft_consensus.cc:2463] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ebb149f387f469fbb3731becb98426f in term 0.
I20250111 02:58:18.766445 21170 leader_election.cc:304] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21; no voters:
I20250111 02:58:18.767618 21230 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:58:18.767993 21230 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:58:18.768309 21230 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:18.776160 21230 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:18.785339 21230 leader_election.cc:290] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 1 election: Requested vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029)
I20250111 02:58:18.788961 21119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21"
I20250111 02:58:18.789768 21119 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:18.799417 21119 raft_consensus.cc:2463] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ebb149f387f469fbb3731becb98426f in term 1.
I20250111 02:58:18.800923 21170 leader_election.cc:304] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21; no voters:
I20250111 02:58:18.802114 21230 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 1 FOLLOWER]: Leader election won for term 1
W20250111 02:58:18.804420 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.804855 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:58:18.805333 21230 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 1 LEADER]: Becoming Leader. State: Replica: 7ebb149f387f469fbb3731becb98426f, State: Running, Role: LEADER
I20250111 02:58:18.806764 21230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
W20250111 02:58:18.823177 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.823660 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:58:18.827282 21256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ebb149f387f469fbb3731becb98426f. Latest consensus state: current_term: 1 leader_uuid: "7ebb149f387f469fbb3731becb98426f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
W20250111 02:58:18.841008 21244 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.841768 21244 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:58:18.829011 21256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: This master's current role is: LEADER
I20250111 02:58:18.850962 21261 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:58:18.859261 21261 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:58:18.880090 21267 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250111 02:58:18.885221 21119 raft_consensus.cc:1270] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 1 FOLLOWER]: Refusing update from remote peer 7ebb149f387f469fbb3731becb98426f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:58:18.887318 21256 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:58:18.886708 21071 master_runner.cc:186] Error getting master registration for 127.18.20.126:34627: OK, instance_id { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" instance_seqno: 1736564294489156 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250111 02:58:18.918943 21071 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:58:18.942370 21249 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ebb149f387f469fbb3731becb98426f. Latest consensus state: current_term: 1 leader_uuid: "7ebb149f387f469fbb3731becb98426f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:18.943249 21249 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:18.943979 21268 mvcc.cc:204] Tried to move back new op lower bound from 7112967368216670208 to 7112967367927373824. Current Snapshot: MvccSnapshot[applied={T|T < 7112967368216670208}]
I20250111 02:58:18.944618 21256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ebb149f387f469fbb3731becb98426f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:18.945381 21256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: This master's current role is: LEADER
W20250111 02:58:18.946293 21173 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.20.126:34627 timed out after 0.523s (SENT)
I20250111 02:58:18.952386 21261 catalog_manager.cc:1348] Generated new cluster ID: 3cbb96e3d31f437a849351b045304259
I20250111 02:58:18.952736 21261 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
W20250111 02:58:18.961871 21267 catalog_manager.cc:1559] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:58:18.962415 21267 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:58:18.981138 21249 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ebb149f387f469fbb3731becb98426f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:18.982098 21249 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:18.998404 21261 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:58:19.000736 21261 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:58:19.013499 21267 catalog_manager.cc:1260] Loaded cluster ID: 3cbb96e3d31f437a849351b045304259
I20250111 02:58:19.015038 21267 catalog_manager.cc:1553] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: loading cluster ID for follower catalog manager: success
I20250111 02:58:19.033767 21261 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: Generated new TSK 0
I20250111 02:58:19.034904 21267 catalog_manager.cc:1575] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: acquiring CA information for follower catalog manager: success
I20250111 02:58:19.041610 21267 catalog_manager.cc:1603] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 02:58:19.035041 21261 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:58:19.047134 21261 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 02:58:19.061300 21261 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250111 02:58:19.110072 18513 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
/tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.20.65:0
--local_ip_for_outbound_sockets=127.18.20.65
--webserver_interface=127.18.20.65
--webserver_port=0
--tserver_master_addrs=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--builtin_ntp_servers=127.18.20.84:35279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250111 02:58:19.150247 21070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } attempt: 7
I20250111 02:58:19.203243 21070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Bootstrap starting.
I20250111 02:58:19.227402 21070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:19.230252 21070 log.cc:826] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Log is configured to *not* fsync() on all Append() calls
I20250111 02:58:19.245709 21070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: No bootstrap required, opened a new log
I20250111 02:58:19.309546 21070 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:19.320497 21070 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:19.320984 21070 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3476b1c08aec44598f1c203c2cbcf016, State: Initialized, Role: FOLLOWER
I20250111 02:58:19.322086 21070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:19.362180 21070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:58:19.363763 21043 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:19.364439 21044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3476b1c08aec44598f1c203c2cbcf016"
W20250111 02:58:19.368106 21044 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49660 (request call id 3) took 585 ms (client timeout 714 ms). Trace:
W20250111 02:58:19.368624 21044 rpcz_store.cc:269] 0111 02:58:18.782985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:58:18.783237 (+ 252us) service_pool.cc:225] Handling call
0111 02:58:19.368005 (+584768us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:58:19.362346 21042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3476b1c08aec44598f1c203c2cbcf016" is_pre_election: true
W20250111 02:58:19.372483 21042 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49660 (request call id 5) took 413 ms (client timeout 523 ms). Trace:
W20250111 02:58:19.372963 21042 rpcz_store.cc:269] 0111 02:58:18.958720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:58:18.958878 (+ 158us) service_pool.cc:225] Handling call
0111 02:58:19.371997 (+413119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:58:19.366427 21045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3476b1c08aec44598f1c203c2cbcf016" is_pre_election: true
W20250111 02:58:19.376897 21045 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49660 (request call id 2) took 954 ms (client timeout 523 ms). Trace:
W20250111 02:58:19.378386 21045 rpcz_store.cc:269] 0111 02:58:18.422772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:58:18.422943 (+ 171us) service_pool.cc:225] Handling call
0111 02:58:19.376725 (+953782us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250111 02:58:19.360652 21278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:19.395813 21278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:19.400981 21043 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 1 FOLLOWER]: Refusing update from remote peer 7ebb149f387f469fbb3731becb98426f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:58:19.405712 21230 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250111 02:58:19.451545 21024 tablet.cc:2367] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:58:19.512780 21278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ebb149f387f469fbb3731becb98426f. Latest consensus state: current_term: 1 leader_uuid: "7ebb149f387f469fbb3731becb98426f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:19.512607 21288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ebb149f387f469fbb3731becb98426f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:19.513581 21278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:19.513665 21288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:19.589531 21299 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250111 02:58:19.648165 21299 catalog_manager.cc:1260] Loaded cluster ID: 3cbb96e3d31f437a849351b045304259
I20250111 02:58:19.651661 21299 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: loading cluster ID for follower catalog manager: success
I20250111 02:58:19.672474 21299 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: acquiring CA information for follower catalog manager: success
I20250111 02:58:19.683555 21299 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250111 02:58:19.856772 21275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:58:19.857484 21275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:58:19.858477 21275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250111 02:58:19.911247 21288 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ebb149f387f469fbb3731becb98426f)
I20250111 02:58:19.911871 21288 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:19.915154 21288 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
I20250111 02:58:19.915505 21118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" is_pre_election: true
I20250111 02:58:19.915733 21196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ebb149f387f469fbb3731becb98426f" is_pre_election: true
I20250111 02:58:19.917642 21021 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:19.919009 21288 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250111 02:58:19.928499 21275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:58:19.930127 21275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.20.65
I20250111 02:58:20.002002 21230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ebb149f387f469fbb3731becb98426f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:20.002873 21230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: This master's current role is: LEADER
I20250111 02:58:20.023788 21275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.20.84:35279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.20.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.20.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.20.65
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4120
TSAN enabled
I20250111 02:58:20.025745 21275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:58:20.028154 21275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:58:20.074328 21310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:20.415145 21315 raft_consensus.cc:491] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ebb149f387f469fbb3731becb98426f)
I20250111 02:58:20.418768 21315 raft_consensus.cc:513] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:20.425788 21315 leader_election.cc:290] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
I20250111 02:58:20.427752 21045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3476b1c08aec44598f1c203c2cbcf016" is_pre_election: true
I20250111 02:58:20.439445 21196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ebb149f387f469fbb3731becb98426f" is_pre_election: true
I20250111 02:58:20.442359 21096 leader_election.cc:304] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf3a224ebe0341df84e1d1d8fdfb0f21; no voters: 3476b1c08aec44598f1c203c2cbcf016, 7ebb149f387f469fbb3731becb98426f
I20250111 02:58:20.443383 21315 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:58:20.959867 21316 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ebb149f387f469fbb3731becb98426f)
I20250111 02:58:20.960671 21316 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:20.963766 21118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" is_pre_election: true
I20250111 02:58:20.964527 21118 raft_consensus.cc:2463] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3476b1c08aec44598f1c203c2cbcf016 in term 1.
I20250111 02:58:20.965845 21019 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016, bf3a224ebe0341df84e1d1d8fdfb0f21; no voters:
I20250111 02:58:20.966156 21196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ebb149f387f469fbb3731becb98426f" is_pre_election: true
I20250111 02:58:20.965102 21316 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
I20250111 02:58:20.978152 21316 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 02:58:20.979594 21316 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7ebb149f387f469fbb3731becb98426f)
I20250111 02:58:20.980100 21316 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:58:20.994796 21316 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:21.000779 21118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21"
I20250111 02:58:21.001427 21316 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 2 election: Requested vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
I20250111 02:58:21.002503 21196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ebb149f387f469fbb3731becb98426f"
I20250111 02:58:21.003886 21021 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:21.004925 21317 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250111 02:58:21.016242 21045 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 2 FOLLOWER]: Rejecting Update request from peer 7ebb149f387f469fbb3731becb98426f for earlier term 1. Current term is 2. Ops: []
I20250111 02:58:21.018043 21318 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 02:58:21.032603 21319 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:58:21.032996 21319 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7ebb149f387f469fbb3731becb98426f, State: Running, Role: LEADER
I20250111 02:58:21.033905 21319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:21.035339 21319 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:58:21.056751 21244 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 02:58:21.503713 21319 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:21.504223 21319 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:21.506254 21319 leader_election.cc:290] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029)
I20250111 02:58:21.508504 21118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" is_pre_election: true
I20250111 02:58:21.509176 21118 raft_consensus.cc:2463] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ebb149f387f469fbb3731becb98426f in term 1.
I20250111 02:58:21.509364 21045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3476b1c08aec44598f1c203c2cbcf016" is_pre_election: true
I20250111 02:58:21.510040 21045 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ebb149f387f469fbb3731becb98426f in term 2.
I20250111 02:58:21.512022 21170 leader_election.cc:304] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21; no voters:
I20250111 02:58:21.515326 21319 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250111 02:58:21.515794 21319 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:58:21.516188 21319 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:58:21.533378 21319 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:21.540174 21045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3476b1c08aec44598f1c203c2cbcf016"
I20250111 02:58:21.540820 21045 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:58:21.548944 21045 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ebb149f387f469fbb3731becb98426f in term 3.
I20250111 02:58:21.551596 21173 leader_election.cc:304] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016, 7ebb149f387f469fbb3731becb98426f; no voters:
I20250111 02:58:21.555934 21321 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 3 FOLLOWER]: Leader election won for term 3
I20250111 02:58:21.559603 21118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21"
I20250111 02:58:21.560269 21118 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 1 FOLLOWER]: Advancing to term 3
I20250111 02:58:21.567647 21321 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 3 LEADER]: Becoming Leader. State: Replica: 7ebb149f387f469fbb3731becb98426f, State: Running, Role: LEADER
I20250111 02:58:21.568590 21321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:21.569859 21118 raft_consensus.cc:2463] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ebb149f387f469fbb3731becb98426f in term 3.
I20250111 02:58:21.571758 21319 leader_election.cc:290] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 3 election: Requested vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029)
I20250111 02:58:21.576318 21320 raft_consensus.cc:491] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:21.581029 21320 raft_consensus.cc:513] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:21.586596 21322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ebb149f387f469fbb3731becb98426f. Latest consensus state: current_term: 3 leader_uuid: "7ebb149f387f469fbb3731becb98426f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:21.587090 21045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3476b1c08aec44598f1c203c2cbcf016" is_pre_election: true
I20250111 02:58:21.587553 21322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: This master's current role is: LEADER
I20250111 02:58:21.587776 21045 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf3a224ebe0341df84e1d1d8fdfb0f21 in term 3.
I20250111 02:58:21.589262 21096 leader_election.cc:304] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016, bf3a224ebe0341df84e1d1d8fdfb0f21; no voters:
I20250111 02:58:21.597858 21196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ebb149f387f469fbb3731becb98426f" is_pre_election: true
I20250111 02:58:21.601244 21320 leader_election.cc:290] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
I20250111 02:58:21.607748 21320 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250111 02:58:21.608964 21320 raft_consensus.cc:491] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:58:21.610221 21320 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:58:21.623528 21324 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:58:21.631669 21320 raft_consensus.cc:513] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:21.639942 21045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3476b1c08aec44598f1c203c2cbcf016"
I20250111 02:58:21.640606 21045 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:58:21.648068 21045 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf3a224ebe0341df84e1d1d8fdfb0f21 in term 4.
I20250111 02:58:21.650786 21096 leader_election.cc:304] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016, bf3a224ebe0341df84e1d1d8fdfb0f21; no voters:
I20250111 02:58:21.651948 21325 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 4 FOLLOWER]: Leader election won for term 4
I20250111 02:58:21.661795 21320 leader_election.cc:290] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 4 election: Requested vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
I20250111 02:58:21.674636 21325 raft_consensus.cc:695] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 4 LEADER]: Becoming Leader. State: Replica: bf3a224ebe0341df84e1d1d8fdfb0f21, State: Running, Role: LEADER
I20250111 02:58:21.676697 21325 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:21.695102 21196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7ebb149f387f469fbb3731becb98426f"
I20250111 02:58:21.705608 21324 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:58:21.710718 21324 catalog_manager.cc:1260] Loaded cluster ID: 3cbb96e3d31f437a849351b045304259
I20250111 02:58:21.714486 21320 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf3a224ebe0341df84e1d1d8fdfb0f21. Latest consensus state: current_term: 4 leader_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:21.716500 21320 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: This master's current role is: LEADER
I20250111 02:58:21.722927 21324 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
W20250111 02:58:20.083837 21313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:58:21.733131 21311 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1662 milliseconds
I20250111 02:58:21.733345 21275 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250111 02:58:21.734802 21328 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
W20250111 02:58:20.079684 21309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:21.740778 21275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:58:21.744083 21324 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:58:21.752377 21275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:58:21.753870 21275 hybrid_clock.cc:648] HybridClock initialized: now 1736564301753825 us; error 55 us; skew 500 ppm
I20250111 02:58:21.755030 21275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:58:21.761797 21328 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:58:21.762470 21324 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: Loaded TSK: 0
I20250111 02:58:21.763895 21324 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:58:21.773658 21328 catalog_manager.cc:1260] Loaded cluster ID: 3cbb96e3d31f437a849351b045304259
I20250111 02:58:21.778483 21324 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 02:58:21.778486 21275 webserver.cc:458] Webserver started at http://127.18.20.65:36207/ using document root <none> and password file <none>
I20250111 02:58:21.779958 21275 fs_manager.cc:362] Metadata directory not provided
I20250111 02:58:21.780268 21275 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:58:21.780392 21328 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:58:21.781795 21275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:58:21.787560 21328 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:58:21.792903 21324 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250111 02:58:21.793304 21275 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/data/instance:
uuid: "2754981a32a44bf889c5d2a226e013da"
format_stamp: "Formatted at 2025-01-11 02:58:21 on dist-test-slave-09x0"
I20250111 02:58:21.795090 21275 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal/instance:
uuid: "2754981a32a44bf889c5d2a226e013da"
format_stamp: "Formatted at 2025-01-11 02:58:21 on dist-test-slave-09x0"
I20250111 02:58:21.804160 21328 catalog_manager.cc:5910] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: Loaded TSK: 0
I20250111 02:58:21.811731 21275 fs_manager.cc:696] Time spent creating directory manager: real 0.016s user 0.012s sys 0.001s
I20250111 02:58:21.811031 21328 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:58:21.818754 21328 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 02:58:21.828042 21333 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:58:21.829535 21275 fs_manager.cc:730] Time spent opening block manager: real 0.012s user 0.006s sys 0.004s
I20250111 02:58:21.829985 21275 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal
uuid: "2754981a32a44bf889c5d2a226e013da"
format_stamp: "Formatted at 2025-01-11 02:58:21 on dist-test-slave-09x0"
I20250111 02:58:21.830457 21275 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:58:21.833508 21328 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250111 02:58:21.928234 21275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:58:21.930486 21275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:58:21.931172 21275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:58:21.934878 21275 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:58:21.942575 21275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:58:21.942907 21275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:58:21.943264 21275 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:58:21.943491 21275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:58:22.016373 21118 raft_consensus.cc:1235] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 4 LEADER]: Rejecting Update request from peer 7ebb149f387f469fbb3731becb98426f for earlier term 3. Current term is 4. Ops: []
I20250111 02:58:22.018853 21322 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250111 02:58:22.019883 21322 consensus_queue.cc:1225] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [LEADER]: Peer bf3a224ebe0341df84e1d1d8fdfb0f21 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250111 02:58:22.020380 21322 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 3 LEADER]: Stepping down as leader of term 3
I20250111 02:58:22.020727 21322 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7ebb149f387f469fbb3731becb98426f, State: Running, Role: LEADER
I20250111 02:58:22.022040 21322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:22.024574 21322 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:58:22.263880 21196 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Refusing update from remote peer bf3a224ebe0341df84e1d1d8fdfb0f21: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20250111 02:58:22.264289 21196 pending_rounds.cc:77] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: Aborting all ops after (but not including) 4
I20250111 02:58:22.264572 21196 pending_rounds.cc:99] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20250111 02:58:22.264837 21196 raft_consensus.cc:2883] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250111 02:58:22.267266 21418 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 02:58:22.273128 21322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf3a224ebe0341df84e1d1d8fdfb0f21. Latest consensus state: current_term: 4 leader_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:22.273941 21322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:22.281523 21045 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Refusing update from remote peer bf3a224ebe0341df84e1d1d8fdfb0f21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250111 02:58:22.290982 21418 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 02:58:22.321631 21322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:22.322389 21322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:22.331049 21421 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:22.337219 21421 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: This master's current role is: LEADER
I20250111 02:58:22.353052 21424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf3a224ebe0341df84e1d1d8fdfb0f21. Latest consensus state: current_term: 4 leader_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:22.355998 21424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:22.359648 21432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:22.360644 21432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:22.438094 21455 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.20.65:34021 every 8 connection(s)
I20250111 02:58:22.438745 21275 rpc_server.cc:307] RPC server started. Bound to: 127.18.20.65:34021
I20250111 02:58:22.442452 21275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/data/info.pb
I20250111 02:58:22.452029 18513 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu as pid 21275
I20250111 02:58:22.452926 18513 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal/instance
I20250111 02:58:22.470394 18513 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
/tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.20.66:0
--local_ip_for_outbound_sockets=127.18.20.66
--webserver_interface=127.18.20.66
--webserver_port=0
--tserver_master_addrs=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--builtin_ntp_servers=127.18.20.84:35279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250111 02:58:22.543548 21457 heartbeater.cc:346] Connected to a master server at 127.18.20.125:35029
I20250111 02:58:22.544085 21457 heartbeater.cc:463] Registering TS with master...
I20250111 02:58:22.545506 21457 heartbeater.cc:510] Master 127.18.20.125:35029 requested a full tablet report, sending...
I20250111 02:58:22.548835 21109 ts_manager.cc:194] Registered new tserver with Master: 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021)
I20250111 02:58:22.551936 21109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.20.65:55213
I20250111 02:58:22.555977 21456 heartbeater.cc:346] Connected to a master server at 127.18.20.124:36707
I20250111 02:58:22.556414 21456 heartbeater.cc:463] Registering TS with master...
I20250111 02:58:22.557148 21456 heartbeater.cc:510] Master 127.18.20.124:36707 requested a full tablet report, sending...
I20250111 02:58:22.558903 21462 heartbeater.cc:346] Connected to a master server at 127.18.20.126:34627
I20250111 02:58:22.559325 21462 heartbeater.cc:463] Registering TS with master...
I20250111 02:58:22.560014 21462 heartbeater.cc:510] Master 127.18.20.126:34627 requested a full tablet report, sending...
I20250111 02:58:22.560582 21186 ts_manager.cc:194] Registered new tserver with Master: 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021)
I20250111 02:58:22.563907 21035 ts_manager.cc:194] Registered new tserver with Master: 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021)
I20250111 02:58:22.793973 21322 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:22.794567 21322 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:22.796665 21322 leader_election.cc:290] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029)
I20250111 02:58:22.799924 21045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3476b1c08aec44598f1c203c2cbcf016" is_pre_election: true
I20250111 02:58:22.799890 21118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" is_pre_election: true
I20250111 02:58:22.805243 21173 leader_election.cc:304] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ebb149f387f469fbb3731becb98426f; no voters: 3476b1c08aec44598f1c203c2cbcf016, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:22.807850 21322 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250111 02:58:22.816792 21432 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:22.817317 21432 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:22.820984 21118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" is_pre_election: true
I20250111 02:58:22.823475 21432 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
I20250111 02:58:22.825078 21196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7ebb149f387f469fbb3731becb98426f" is_pre_election: true
I20250111 02:58:22.826952 21021 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:22.828969 21432 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250111 02:58:22.936831 21421 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:22.941365 21421 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: This master's current role is: LEADER
W20250111 02:58:22.986855 21465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:58:22.987437 21465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:58:22.988107 21465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:58:23.032207 21465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:58:23.033874 21465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.20.66
I20250111 02:58:23.056669 21457 heartbeater.cc:502] Master 127.18.20.125:35029 was elected leader, sending a full tablet report...
I20250111 02:58:23.077778 21465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.20.84:35279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.20.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.20.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.20.66
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4120
TSAN enabled
I20250111 02:58:23.079480 21465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:58:23.081388 21465 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:58:23.101930 21477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:23.328307 21482 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:23.328748 21482 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:23.332669 21045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3476b1c08aec44598f1c203c2cbcf016" is_pre_election: true
I20250111 02:58:23.333698 21482 leader_election.cc:290] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029)
I20250111 02:58:23.334309 21118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" is_pre_election: true
I20250111 02:58:23.335636 21170 leader_election.cc:304] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ebb149f387f469fbb3731becb98426f; no voters: 3476b1c08aec44598f1c203c2cbcf016, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:23.337353 21482 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250111 02:58:23.105239 21479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:23.106096 21465 server_base.cc:1034] running on GCE node
W20250111 02:58:23.104072 21476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:24.339885 21465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:58:24.342244 21465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:58:24.343657 21465 hybrid_clock.cc:648] HybridClock initialized: now 1736564304343622 us; error 41 us; skew 500 ppm
I20250111 02:58:24.344543 21465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:58:24.357148 21465 webserver.cc:458] Webserver started at http://127.18.20.66:36169/ using document root <none> and password file <none>
I20250111 02:58:24.358158 21465 fs_manager.cc:362] Metadata directory not provided
I20250111 02:58:24.358368 21465 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:58:24.358799 21465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:58:24.364612 21465 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/data/instance:
uuid: "b6ee352ce89c48d38b467042c4c3f9fc"
format_stamp: "Formatted at 2025-01-11 02:58:24 on dist-test-slave-09x0"
I20250111 02:58:24.365926 21465 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal/instance:
uuid: "b6ee352ce89c48d38b467042c4c3f9fc"
format_stamp: "Formatted at 2025-01-11 02:58:24 on dist-test-slave-09x0"
I20250111 02:58:24.374059 21465 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20250111 02:58:24.380334 21487 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:58:24.381731 21465 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250111 02:58:24.382114 21465 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal
uuid: "b6ee352ce89c48d38b467042c4c3f9fc"
format_stamp: "Formatted at 2025-01-11 02:58:24 on dist-test-slave-09x0"
I20250111 02:58:24.382445 21465 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:58:24.387873 21488 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:24.388417 21488 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:24.390167 21488 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
I20250111 02:58:24.391108 21118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" is_pre_election: true
I20250111 02:58:24.391489 21196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7ebb149f387f469fbb3731becb98426f" is_pre_election: true
I20250111 02:58:24.393291 21021 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:24.395591 21488 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250111 02:58:24.464407 21465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:58:24.466082 21465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:58:24.466548 21465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:58:24.469403 21465 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:58:24.473932 21465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:58:24.474164 21465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:58:24.474375 21465 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:58:24.474521 21465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:58:24.636099 21465 rpc_server.cc:307] RPC server started. Bound to: 127.18.20.66:45313
I20250111 02:58:24.636274 21600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.20.66:45313 every 8 connection(s)
I20250111 02:58:24.638798 21465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/data/info.pb
I20250111 02:58:24.645857 18513 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu as pid 21465
I20250111 02:58:24.646520 18513 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal/instance
I20250111 02:58:24.669894 18513 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
/tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.20.67:0
--local_ip_for_outbound_sockets=127.18.20.67
--webserver_interface=127.18.20.67
--webserver_port=0
--tserver_master_addrs=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--builtin_ntp_servers=127.18.20.84:35279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250111 02:58:24.690424 21602 heartbeater.cc:346] Connected to a master server at 127.18.20.125:35029
I20250111 02:58:24.691097 21602 heartbeater.cc:463] Registering TS with master...
I20250111 02:58:24.691861 21601 heartbeater.cc:346] Connected to a master server at 127.18.20.124:36707
I20250111 02:58:24.692325 21601 heartbeater.cc:463] Registering TS with master...
I20250111 02:58:24.692878 21602 heartbeater.cc:510] Master 127.18.20.125:35029 requested a full tablet report, sending...
I20250111 02:58:24.693409 21601 heartbeater.cc:510] Master 127.18.20.124:36707 requested a full tablet report, sending...
I20250111 02:58:24.697042 21186 ts_manager.cc:194] Registered new tserver with Master: b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313)
I20250111 02:58:24.697002 21109 ts_manager.cc:194] Registered new tserver with Master: b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313)
I20250111 02:58:24.699962 21109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.20.66:42077
I20250111 02:58:24.704707 21603 heartbeater.cc:346] Connected to a master server at 127.18.20.126:34627
I20250111 02:58:24.705104 21603 heartbeater.cc:463] Registering TS with master...
I20250111 02:58:24.705690 21603 heartbeater.cc:510] Master 127.18.20.126:34627 requested a full tablet report, sending...
I20250111 02:58:24.708220 21612 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:24.708649 21035 ts_manager.cc:194] Registered new tserver with Master: b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313)
I20250111 02:58:24.708611 21612 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:24.710163 21612 leader_election.cc:290] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029)
I20250111 02:58:24.711474 21118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" is_pre_election: true
I20250111 02:58:24.711464 21045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3476b1c08aec44598f1c203c2cbcf016" is_pre_election: true
I20250111 02:58:24.713162 21173 leader_election.cc:304] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ebb149f387f469fbb3731becb98426f; no voters: 3476b1c08aec44598f1c203c2cbcf016, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:24.714115 21612 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250111 02:58:24.941134 21614 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:24.941645 21614 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:24.943245 21614 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
I20250111 02:58:24.944312 21196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7ebb149f387f469fbb3731becb98426f" is_pre_election: true
I20250111 02:58:24.944957 21118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" is_pre_election: true
I20250111 02:58:24.946887 21019 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:24.947614 21614 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250111 02:58:25.041774 21611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:58:25.042331 21611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:58:25.042948 21611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:58:25.078578 21611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:58:25.079962 21611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.20.67
I20250111 02:58:25.119186 21611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.20.84:35279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.20.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.20.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.20.67
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4120
TSAN enabled
I20250111 02:58:25.120560 21611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:58:25.122354 21611 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:58:25.143810 21621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:25.204850 21602 heartbeater.cc:502] Master 127.18.20.125:35029 was elected leader, sending a full tablet report...
I20250111 02:58:25.290297 21626 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:25.291054 21626 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:25.294240 21045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3476b1c08aec44598f1c203c2cbcf016" is_pre_election: true
I20250111 02:58:25.295137 21118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" is_pre_election: true
I20250111 02:58:25.295974 21626 leader_election.cc:290] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029)
I20250111 02:58:25.296483 21170 leader_election.cc:304] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ebb149f387f469fbb3731becb98426f; no voters: 3476b1c08aec44598f1c203c2cbcf016, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:25.297858 21626 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250111 02:58:25.960613 21627 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:25.961048 21627 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:25.962752 21627 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
I20250111 02:58:25.963675 21118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" is_pre_election: true
I20250111 02:58:25.964264 21196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7ebb149f387f469fbb3731becb98426f" is_pre_election: true
I20250111 02:58:25.966547 21021 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:25.967823 21627 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250111 02:58:26.294116 21452 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0060 after lost signal to thread 21301
W20250111 02:58:26.295105 21452 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0d40 after lost signal to thread 21455
W20250111 02:58:25.145140 21623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:58:25.143949 21620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:25.148947 21611 server_base.cc:1034] running on GCE node
I20250111 02:58:26.558722 21611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:58:26.562202 21611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:58:26.563827 21611 hybrid_clock.cc:648] HybridClock initialized: now 1736564306563788 us; error 45 us; skew 500 ppm
I20250111 02:58:26.565135 21611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:58:26.573956 21611 webserver.cc:458] Webserver started at http://127.18.20.67:39877/ using document root <none> and password file <none>
I20250111 02:58:26.575459 21611 fs_manager.cc:362] Metadata directory not provided
I20250111 02:58:26.575789 21611 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:58:26.576413 21611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:58:26.585250 21611 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/data/instance:
uuid: "6830bca1186e4084868808884170cc33"
format_stamp: "Formatted at 2025-01-11 02:58:26 on dist-test-slave-09x0"
I20250111 02:58:26.587103 21611 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal/instance:
uuid: "6830bca1186e4084868808884170cc33"
format_stamp: "Formatted at 2025-01-11 02:58:26 on dist-test-slave-09x0"
I20250111 02:58:26.600428 21611 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.007s sys 0.004s
I20250111 02:58:26.607998 21632 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:58:26.609338 21611 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20250111 02:58:26.609706 21611 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal
uuid: "6830bca1186e4084868808884170cc33"
format_stamp: "Formatted at 2025-01-11 02:58:26 on dist-test-slave-09x0"
I20250111 02:58:26.610065 21611 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:58:26.682685 21611 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:58:26.685032 21611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:58:26.685694 21611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:58:26.689353 21611 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:58:26.696141 21611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:58:26.696496 21611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:58:26.696832 21611 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:58:26.697062 21611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:58:26.890523 21611 rpc_server.cc:307] RPC server started. Bound to: 127.18.20.67:43789
I20250111 02:58:26.890698 21745 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.20.67:43789 every 8 connection(s)
I20250111 02:58:26.893774 21611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/data/info.pb
I20250111 02:58:26.898519 18513 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu as pid 21611
I20250111 02:58:26.899046 18513 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal/instance
I20250111 02:58:26.972306 21746 heartbeater.cc:346] Connected to a master server at 127.18.20.124:36707
I20250111 02:58:26.972888 21746 heartbeater.cc:463] Registering TS with master...
I20250111 02:58:26.974491 21746 heartbeater.cc:510] Master 127.18.20.124:36707 requested a full tablet report, sending...
I20250111 02:58:26.974773 21750 heartbeater.cc:346] Connected to a master server at 127.18.20.126:34627
I20250111 02:58:26.975121 21750 heartbeater.cc:463] Registering TS with master...
I20250111 02:58:26.976121 21750 heartbeater.cc:510] Master 127.18.20.126:34627 requested a full tablet report, sending...
I20250111 02:58:26.978859 21186 ts_manager.cc:194] Registered new tserver with Master: 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:26.979635 21034 ts_manager.cc:194] Registered new tserver with Master: 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:26.980453 21747 heartbeater.cc:346] Connected to a master server at 127.18.20.125:35029
I20250111 02:58:26.980788 21747 heartbeater.cc:463] Registering TS with master...
I20250111 02:58:26.981711 21747 heartbeater.cc:510] Master 127.18.20.125:35029 requested a full tablet report, sending...
I20250111 02:58:26.984220 21108 ts_manager.cc:194] Registered new tserver with Master: 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:26.986835 21108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.20.67:33157
I20250111 02:58:26.988982 18513 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250111 02:58:27.051609 21107 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59588:
name: "default.Table_95e9bf8a21a4474781705bea1e0c7b91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:27.053391 21108 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59588:
name: "default.Table_836e7aa15e09458c9659cb4a8b7d352a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:27.050820 21109 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59588:
name: "default.Table_15ebf4fa41db452c957f7aa50ddfc192"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:58:27.063712 21107 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_95e9bf8a21a4474781705bea1e0c7b91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:58:27.069890 21106 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59588:
name: "default.Table_46ab12e3d44441d982fda4c1aba20a2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:27.142158 21778 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:27.143023 21778 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:27.148671 21045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3476b1c08aec44598f1c203c2cbcf016" is_pre_election: true
I20250111 02:58:27.149323 21778 leader_election.cc:290] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029)
I20250111 02:58:27.150609 21118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" is_pre_election: true
I20250111 02:58:27.154451 21170 leader_election.cc:304] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ebb149f387f469fbb3731becb98426f; no voters: 3476b1c08aec44598f1c203c2cbcf016, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:27.155854 21778 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250111 02:58:27.167809 18513 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu with pid 21071
W20250111 02:58:27.209260 21756 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.20.125:35029)
W20250111 02:58:27.273474 21602 heartbeater.cc:643] Failed to heartbeat to 127.18.20.125:35029 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 02:58:27.396399 21782 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:27.396935 21782 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:27.405232 21196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7ebb149f387f469fbb3731becb98426f" is_pre_election: true
I20250111 02:58:27.406170 21196 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3476b1c08aec44598f1c203c2cbcf016 in term 4.
I20250111 02:58:27.407790 21021 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016, 7ebb149f387f469fbb3731becb98426f; no voters:
I20250111 02:58:27.413905 21782 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
I20250111 02:58:27.414155 21784 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250111 02:58:27.419427 21784 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Starting leader election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:27.419983 21784 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 4 FOLLOWER]: Advancing to term 5
W20250111 02:58:27.414420 21019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20250111 02:58:27.433092 21784 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:27.441416 21196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7ebb149f387f469fbb3731becb98426f"
I20250111 02:58:27.442062 21196 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 4 FOLLOWER]: Advancing to term 5
W20250111 02:58:27.446470 21019 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 02:58:27.446966 21784 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 5 election: Requested vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
I20250111 02:58:27.450956 21196 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3476b1c08aec44598f1c203c2cbcf016 in term 5.
W20250111 02:58:27.454056 21019 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 02:58:27.457093 21021 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016, 7ebb149f387f469fbb3731becb98426f; no voters: bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:27.457942 21784 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 5 FOLLOWER]: Leader election won for term 5
I20250111 02:58:27.458479 21784 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 5 LEADER]: Becoming Leader. State: Replica: 3476b1c08aec44598f1c203c2cbcf016, State: Running, Role: LEADER
I20250111 02:58:27.459601 21784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:27.465968 21782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3476b1c08aec44598f1c203c2cbcf016. Latest consensus state: current_term: 5 leader_uuid: "3476b1c08aec44598f1c203c2cbcf016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:27.466807 21782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: This master's current role is: LEADER
I20250111 02:58:27.472095 21787 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:58:27.482404 21787 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:58:27.492013 21747 heartbeater.cc:502] Master 127.18.20.125:35029 was elected leader, sending a full tablet report...
I20250111 02:58:27.492245 21787 catalog_manager.cc:1260] Loaded cluster ID: 3cbb96e3d31f437a849351b045304259
I20250111 02:58:27.492503 21787 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:58:27.502458 21787 catalog_manager.cc:1505] Loading token signing keys...
W20250111 02:58:27.505729 21747 heartbeater.cc:643] Failed to heartbeat to 127.18.20.125:35029 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 02:58:27.508698 21787 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Loaded TSK: 0
I20250111 02:58:27.519389 21787 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:58:27.524870 21787 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 02:58:27.529834 21787 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
W20250111 02:58:27.613801 21457 heartbeater.cc:643] Failed to heartbeat to 127.18.20.125:35029 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 02:58:27.555987 21033 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49668:
name: "default.Table_15ebf4fa41db452c957f7aa50ddfc192"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:27.557534 21035 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49668:
name: "default.Table_46ab12e3d44441d982fda4c1aba20a2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:27.549654 21032 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49668:
name: "default.Table_836e7aa15e09458c9659cb4a8b7d352a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:58:27.642805 21035 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_46ab12e3d44441d982fda4c1aba20a2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 02:58:27.647735 21033 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_15ebf4fa41db452c957f7aa50ddfc192 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:58:27.549482 21034 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49668:
name: "default.Table_95e9bf8a21a4474781705bea1e0c7b91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:27.844267 21196 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Refusing update from remote peer 3476b1c08aec44598f1c203c2cbcf016: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250111 02:58:27.846338 21782 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250111 02:58:27.858640 18513 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
/tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.125:35029
--webserver_interface=127.18.20.125
--webserver_port=38137
--builtin_ntp_servers=127.18.20.84:35279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--hive_metastore_uris=thrift://127.0.0.1:41043
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250111 02:58:27.884837 21791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3476b1c08aec44598f1c203c2cbcf016. Latest consensus state: current_term: 5 leader_uuid: "3476b1c08aec44598f1c203c2cbcf016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:27.909319 21791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:27.915918 21791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "3476b1c08aec44598f1c203c2cbcf016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:27.918104 21791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: This master's current role is: FOLLOWER
W20250111 02:58:27.930956 21019 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 -> Peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Couldn't send request to peer bf3a224ebe0341df84e1d1d8fdfb0f21. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:58:27.932760 21782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "3476b1c08aec44598f1c203c2cbcf016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:27.933629 21782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: This master's current role is: LEADER
I20250111 02:58:28.085911 21033 catalog_manager.cc:2149] created HMS catalog entry for table [id=22ba37c4c77d4f03981e24ae0ef1adcb]
I20250111 02:58:28.109020 21462 heartbeater.cc:502] Master 127.18.20.126:34627 was elected leader, sending a full tablet report...
I20250111 02:58:28.122587 21033 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49668:
table { table_name: "default.Table_15ebf4fa41db452c957f7aa50ddfc192" } new_table_name: "default.Table_7e8389e7a1b745a8898f6a0862a97327" modify_external_catalogs: true
I20250111 02:58:28.184643 21035 catalog_manager.cc:2149] created HMS catalog entry for table [id=9089e2a62ec84b4b86a93cbc757190d4]
W20250111 02:58:28.187160 21032 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_836e7aa15e09458c9659cb4a8b7d352a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
F20250111 02:58:28.210191 21298 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_95e9bf8a21a4474781705bea1e0c7b91
*** Check failure stack trace: ***
I20250111 02:58:28.225960 21035 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49668:
table { table_name: "default.Table_46ab12e3d44441d982fda4c1aba20a2d" } new_table_name: "default.Table_77a9864685c248e9bb93b3b1f75adb6e" modify_external_catalogs: true
I20250111 02:58:28.258515 21536 tablet_service.cc:1467] Processing CreateTablet for tablet e945d233bffc4251bcd74340c3e6f8c6 (DEFAULT_TABLE table=default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f79725bc950 google::LogMessage::Flush() at ??:0
I20250111 02:58:28.260922 21536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e945d233bffc4251bcd74340c3e6f8c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:28.268165 21603 heartbeater.cc:502] Master 127.18.20.126:34627 was elected leader, sending a full tablet report...
I20250111 02:58:28.279160 21681 tablet_service.cc:1467] Processing CreateTablet for tablet e945d233bffc4251bcd74340c3e6f8c6 (DEFAULT_TABLE table=default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:28.281280 21681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e945d233bffc4251bcd74340c3e6f8c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:28.301846 21381 tablet_service.cc:1467] Processing CreateTablet for tablet e945d233bffc4251bcd74340c3e6f8c6 (DEFAULT_TABLE table=default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:28.303102 21680 tablet_service.cc:1467] Processing CreateTablet for tablet 55be07345aff496c8cc910f921254ded (DEFAULT_TABLE table=default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:28.303591 21380 tablet_service.cc:1467] Processing CreateTablet for tablet 55be07345aff496c8cc910f921254ded (DEFAULT_TABLE table=default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:28.304410 21680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55be07345aff496c8cc910f921254ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:28.305231 21380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55be07345aff496c8cc910f921254ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:28.307740 21381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e945d233bffc4251bcd74340c3e6f8c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:28.323493 21031 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49668:
name: "default.Table_b371b664c5b043be8b2f0353a9e57e8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:28.340803 21535 tablet_service.cc:1467] Processing CreateTablet for tablet 55be07345aff496c8cc910f921254ded (DEFAULT_TABLE table=default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:28.342029 21535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55be07345aff496c8cc910f921254ded. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f79725bc6bf google::LogMessage::~LogMessage() at ??:0
I20250111 02:58:28.380656 21809 tablet_bootstrap.cc:492] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33: Bootstrap starting.
I20250111 02:58:28.396569 21810 tablet_bootstrap.cc:492] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc: Bootstrap starting.
I20250111 02:58:28.410524 21032 catalog_manager.cc:2149] created HMS catalog entry for table [id=1e441333efa8493fac8f81eba63d7e7f]
I20250111 02:58:28.419117 21030 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49668:
name: "default.Table_0939bec9a16f450fa14248fcc1cd2fc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:28.423097 21809 tablet_bootstrap.cc:654] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:28.425778 21809 log.cc:826] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33: Log is configured to *not* fsync() on all Append() calls
I20250111 02:58:28.448500 21032 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49668:
table { table_name: "default.Table_836e7aa15e09458c9659cb4a8b7d352a" } modify_external_catalogs: true
I20250111 02:58:28.449342 21032 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49668:
table { table_name: "default.Table_836e7aa15e09458c9659cb4a8b7d352a" } modify_external_catalogs: true
I20250111 02:58:28.449903 21029 master_service.cc:946] ReplaceTablet: received request to replace tablet 55be07345aff496c8cc910f921254ded from {username='slave'} at 127.0.0.1:49668
I20250111 02:58:28.455109 21028 master_service.cc:946] ReplaceTablet: received request to replace tablet 55be07345aff496c8cc910f921254ded from {username='slave'} at 127.0.0.1:49668
I20250111 02:58:28.455216 21812 tablet_bootstrap.cc:492] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da: Bootstrap starting.
I20250111 02:58:28.456276 21810 tablet_bootstrap.cc:654] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:28.465629 21810 log.cc:826] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc: Log is configured to *not* fsync() on all Append() calls
I20250111 02:58:28.472152 21810 tablet_bootstrap.cc:492] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc: No bootstrap required, opened a new log
I20250111 02:58:28.473017 21810 ts_tablet_manager.cc:1397] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc: Time spent bootstrapping tablet: real 0.077s user 0.018s sys 0.000s
I20250111 02:58:28.482018 21381 tablet_service.cc:1467] Processing CreateTablet for tablet 3ca2aca3ca0447d8af6941450a537e36 (DEFAULT_TABLE table=default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:28.483263 21381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca2aca3ca0447d8af6941450a537e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:28.482959 21681 tablet_service.cc:1467] Processing CreateTablet for tablet 3ca2aca3ca0447d8af6941450a537e36 (DEFAULT_TABLE table=default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:28.484283 21681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca2aca3ca0447d8af6941450a537e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:28.486202 21809 tablet_bootstrap.cc:492] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33: No bootstrap required, opened a new log
I20250111 02:58:28.486809 21809 ts_tablet_manager.cc:1397] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33: Time spent bootstrapping tablet: real 0.107s user 0.021s sys 0.004s
I20250111 02:58:28.488802 21535 tablet_service.cc:1467] Processing CreateTablet for tablet 3ca2aca3ca0447d8af6941450a537e36 (DEFAULT_TABLE table=default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:28.490139 21535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca2aca3ca0447d8af6941450a537e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:28.495558 21750 heartbeater.cc:502] Master 127.18.20.126:34627 was elected leader, sending a full tablet report...
I20250111 02:58:28.504738 21812 tablet_bootstrap.cc:654] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:28.523494 21812 log.cc:826] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da: Log is configured to *not* fsync() on all Append() calls
@ 0x7f797d4b782a kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250111 02:58:28.532068 21029 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Sending DeleteTablet for 3 replicas of tablet 55be07345aff496c8cc910f921254ded
I20250111 02:58:28.533730 21029 catalog_manager.cc:6596] ReplaceTablet: tablet 55be07345aff496c8cc910f921254ded deleted and replaced by tablet caf9a22c859d46e29bb216e6248bfa1b
I20250111 02:58:28.535538 21680 tablet_service.cc:1514] Processing DeleteTablet for tablet 55be07345aff496c8cc910f921254ded with delete_type TABLET_DATA_DELETED (replaced by tablet caf9a22c859d46e29bb216e6248bfa1b) from {username='slave'} at 127.0.0.1:44898
I20250111 02:58:28.542546 21536 tablet_service.cc:1514] Processing DeleteTablet for tablet 55be07345aff496c8cc910f921254ded with delete_type TABLET_DATA_DELETED (replaced by tablet caf9a22c859d46e29bb216e6248bfa1b) from {username='slave'} at 127.0.0.1:52636
I20250111 02:58:28.545312 21380 tablet_service.cc:1514] Processing DeleteTablet for tablet 55be07345aff496c8cc910f921254ded with delete_type TABLET_DATA_DELETED (replaced by tablet caf9a22c859d46e29bb216e6248bfa1b) from {username='slave'} at 127.0.0.1:36052
W20250111 02:58:28.547032 21020 catalog_manager.cc:4852] TS b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313): delete failed for tablet 55be07345aff496c8cc910f921254ded because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55be07345aff496c8cc910f921254ded already in progress: creating tablet
I20250111 02:58:28.560741 21812 tablet_bootstrap.cc:492] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da: No bootstrap required, opened a new log
I20250111 02:58:28.561354 21812 ts_tablet_manager.cc:1397] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da: Time spent bootstrapping tablet: real 0.123s user 0.009s sys 0.010s
I20250111 02:58:28.575219 21810 raft_consensus.cc:357] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.576455 21810 raft_consensus.cc:383] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:28.576843 21810 raft_consensus.cc:738] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Initialized, Role: FOLLOWER
I20250111 02:58:28.581518 21536 tablet_service.cc:1467] Processing CreateTablet for tablet caf9a22c859d46e29bb216e6248bfa1b (DEFAULT_TABLE table=default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:28.582798 21536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf9a22c859d46e29bb216e6248bfa1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:28.583709 21810 consensus_queue.cc:260] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.593995 21810 ts_tablet_manager.cc:1428] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc: Time spent starting tablet: real 0.121s user 0.034s sys 0.011s
I20250111 02:58:28.595295 21810 tablet_bootstrap.cc:492] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc: Bootstrap starting.
I20250111 02:58:28.602680 21810 tablet_bootstrap.cc:654] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:28.609836 21535 tablet_service.cc:1514] Processing DeleteTablet for tablet 55be07345aff496c8cc910f921254ded with delete_type TABLET_DATA_DELETED (replaced by tablet caf9a22c859d46e29bb216e6248bfa1b) from {username='slave'} at 127.0.0.1:52636
I20250111 02:58:28.613013 21809 raft_consensus.cc:357] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.612285 21820 tablet_replica.cc:331] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc: stopping tablet replica
I20250111 02:58:28.614535 21809 raft_consensus.cc:383] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:28.614943 21809 raft_consensus.cc:738] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830bca1186e4084868808884170cc33, State: Initialized, Role: FOLLOWER
I20250111 02:58:28.606673 21679 tablet_service.cc:1467] Processing CreateTablet for tablet caf9a22c859d46e29bb216e6248bfa1b (DEFAULT_TABLE table=default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:28.616708 21381 tablet_service.cc:1467] Processing CreateTablet for tablet caf9a22c859d46e29bb216e6248bfa1b (DEFAULT_TABLE table=default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:28.617014 21679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf9a22c859d46e29bb216e6248bfa1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:28.617878 21381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf9a22c859d46e29bb216e6248bfa1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:28.622676 21820 raft_consensus.cc:2238] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:28.623494 21820 raft_consensus.cc:2267] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:28.623585 21809 consensus_queue.cc:260] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.636324 21809 ts_tablet_manager.cc:1428] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33: Time spent starting tablet: real 0.149s user 0.033s sys 0.009s
W20250111 02:58:28.637921 21022 catalog_manager.cc:4852] TS 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): delete failed for tablet 55be07345aff496c8cc910f921254ded because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55be07345aff496c8cc910f921254ded already in progress: creating tablet
I20250111 02:58:28.639361 21809 tablet_bootstrap.cc:492] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33: Bootstrap starting.
I20250111 02:58:28.648442 21027 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49668:
name: "default.Table_dd8c5f570a5c46a4a461cb9d3c409afe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:58:28.663019 21019 catalog_manager.cc:4852] TS 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): delete failed for tablet 55be07345aff496c8cc910f921254ded because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55be07345aff496c8cc910f921254ded already in progress: creating tablet
I20250111 02:58:28.666987 21820 ts_tablet_manager.cc:1905] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:28.670086 21809 tablet_bootstrap.cc:654] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:28.693693 21810 tablet_bootstrap.cc:492] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc: No bootstrap required, opened a new log
I20250111 02:58:28.694208 21810 ts_tablet_manager.cc:1397] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc: Time spent bootstrapping tablet: real 0.099s user 0.012s sys 0.024s
I20250111 02:58:28.696928 21810 raft_consensus.cc:357] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.697656 21810 raft_consensus.cc:383] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:28.697928 21810 raft_consensus.cc:738] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Initialized, Role: FOLLOWER
I20250111 02:58:28.698588 21810 consensus_queue.cc:260] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
W20250111 02:58:28.702091 21755 tablet.cc:2367] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250111 02:58:28.709218 21608 tablet.cc:2367] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:58:28.701326 21810 ts_tablet_manager.cc:1428] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250111 02:58:28.711625 21810 tablet_bootstrap.cc:492] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc: Bootstrap starting.
I20250111 02:58:28.711498 21809 tablet_bootstrap.cc:492] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33: No bootstrap required, opened a new log
I20250111 02:58:28.714066 21809 ts_tablet_manager.cc:1397] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33: Time spent bootstrapping tablet: real 0.075s user 0.013s sys 0.000s
I20250111 02:58:28.717044 21809 raft_consensus.cc:357] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.718948 21809 raft_consensus.cc:383] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:28.719343 21809 raft_consensus.cc:738] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830bca1186e4084868808884170cc33, State: Initialized, Role: FOLLOWER
I20250111 02:58:28.727819 21810 tablet_bootstrap.cc:654] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:28.727883 21820 ts_tablet_manager.cc:1918] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:28.727454 21809 consensus_queue.cc:260] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.728367 21820 log.cc:1198] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/55be07345aff496c8cc910f921254ded
I20250111 02:58:28.729506 21820 ts_tablet_manager.cc:1939] T 55be07345aff496c8cc910f921254ded P b6ee352ce89c48d38b467042c4c3f9fc: Deleting consensus metadata
I20250111 02:58:28.725050 21812 raft_consensus.cc:357] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.730245 21812 raft_consensus.cc:383] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:28.732474 21812 raft_consensus.cc:738] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Initialized, Role: FOLLOWER
I20250111 02:58:28.732951 21020 catalog_manager.cc:4872] TS b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313): tablet 55be07345aff496c8cc910f921254ded (table default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]) successfully deleted
I20250111 02:58:28.733670 21812 consensus_queue.cc:260] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
@ 0x7f797d4b707c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250111 02:58:28.745540 21679 tablet_service.cc:1514] Processing DeleteTablet for tablet 55be07345aff496c8cc910f921254ded with delete_type TABLET_DATA_DELETED (replaced by tablet caf9a22c859d46e29bb216e6248bfa1b) from {username='slave'} at 127.0.0.1:44898
W20250111 02:58:28.747480 21022 catalog_manager.cc:4852] TS 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): delete failed for tablet 55be07345aff496c8cc910f921254ded because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55be07345aff496c8cc910f921254ded already in progress: creating tablet
I20250111 02:58:28.750464 21029 master_service.cc:946] ReplaceTablet: received request to replace tablet e945d233bffc4251bcd74340c3e6f8c6 from {username='slave'} at 127.0.0.1:49668
I20250111 02:58:28.754290 21810 tablet_bootstrap.cc:492] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc: No bootstrap required, opened a new log
I20250111 02:58:28.760619 21810 ts_tablet_manager.cc:1397] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.004s
I20250111 02:58:28.764277 21809 ts_tablet_manager.cc:1428] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33: Time spent starting tablet: real 0.050s user 0.007s sys 0.000s
I20250111 02:58:28.743743 21812 ts_tablet_manager.cc:1428] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da: Time spent starting tablet: real 0.182s user 0.040s sys 0.006s
W20250111 02:58:28.756505 21466 tablet.cc:2367] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:58:28.771448 21810 raft_consensus.cc:357] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.772163 21810 raft_consensus.cc:383] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:28.772434 21810 raft_consensus.cc:738] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Initialized, Role: FOLLOWER
I20250111 02:58:28.772567 21679 tablet_service.cc:1514] Processing DeleteTablet for tablet 55be07345aff496c8cc910f921254ded with delete_type TABLET_DATA_DELETED (replaced by tablet caf9a22c859d46e29bb216e6248bfa1b) from {username='slave'} at 127.0.0.1:44898
I20250111 02:58:28.774082 21812 tablet_bootstrap.cc:492] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da: Bootstrap starting.
I20250111 02:58:28.774124 21819 tablet_replica.cc:331] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33: stopping tablet replica
I20250111 02:58:28.775049 21819 raft_consensus.cc:2238] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:28.775699 21819 raft_consensus.cc:2267] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:28.779129 21810 consensus_queue.cc:260] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.783843 21810 ts_tablet_manager.cc:1428] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20250111 02:58:28.788497 21810 tablet_bootstrap.cc:492] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc: Bootstrap starting.
I20250111 02:58:28.790421 21029 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Sending DeleteTablet for 3 replicas of tablet e945d233bffc4251bcd74340c3e6f8c6
I20250111 02:58:28.793092 21819 ts_tablet_manager.cc:1905] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:28.795171 21536 tablet_service.cc:1514] Processing DeleteTablet for tablet e945d233bffc4251bcd74340c3e6f8c6 with delete_type TABLET_DATA_DELETED (replaced by tablet dfbc1d2cb1764752a73807376d44452e) from {username='slave'} at 127.0.0.1:52636
I20250111 02:58:28.796361 21820 tablet_replica.cc:331] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc: stopping tablet replica
I20250111 02:58:28.796557 21029 catalog_manager.cc:6596] ReplaceTablet: tablet e945d233bffc4251bcd74340c3e6f8c6 deleted and replaced by tablet dfbc1d2cb1764752a73807376d44452e
I20250111 02:58:28.797120 21820 raft_consensus.cc:2238] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:28.797622 21820 raft_consensus.cc:2267] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:28.797605 21381 tablet_service.cc:1514] Processing DeleteTablet for tablet e945d233bffc4251bcd74340c3e6f8c6 with delete_type TABLET_DATA_DELETED (replaced by tablet dfbc1d2cb1764752a73807376d44452e) from {username='slave'} at 127.0.0.1:36052
I20250111 02:58:28.798571 21821 tablet_replica.cc:331] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da: stopping tablet replica
I20250111 02:58:28.799013 21380 tablet_service.cc:1514] Processing DeleteTablet for tablet e945d233bffc4251bcd74340c3e6f8c6 with delete_type TABLET_DATA_DELETED (replaced by tablet dfbc1d2cb1764752a73807376d44452e) from {username='slave'} at 127.0.0.1:36052
I20250111 02:58:28.799515 21821 raft_consensus.cc:2238] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:28.800130 21821 raft_consensus.cc:2267] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:28.802995 21679 tablet_service.cc:1514] Processing DeleteTablet for tablet e945d233bffc4251bcd74340c3e6f8c6 with delete_type TABLET_DATA_DELETED (replaced by tablet dfbc1d2cb1764752a73807376d44452e) from {username='slave'} at 127.0.0.1:44898
I20250111 02:58:28.805583 21820 ts_tablet_manager.cc:1905] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:28.806046 21821 ts_tablet_manager.cc:1905] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:28.818517 21810 tablet_bootstrap.cc:654] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:28.787483 21812 tablet_bootstrap.cc:654] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:28.824419 21679 tablet_service.cc:1467] Processing CreateTablet for tablet dfbc1d2cb1764752a73807376d44452e (DEFAULT_TABLE table=default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:28.825521 21679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfbc1d2cb1764752a73807376d44452e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:28.826071 21380 tablet_service.cc:1467] Processing CreateTablet for tablet dfbc1d2cb1764752a73807376d44452e (DEFAULT_TABLE table=default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:28.827266 21380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfbc1d2cb1764752a73807376d44452e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:28.830925 21028 master_service.cc:946] ReplaceTablet: received request to replace tablet 55be07345aff496c8cc910f921254ded from {username='slave'} at 127.0.0.1:49668
I20250111 02:58:28.823383 21536 tablet_service.cc:1467] Processing CreateTablet for tablet dfbc1d2cb1764752a73807376d44452e (DEFAULT_TABLE table=default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:28.851900 21536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfbc1d2cb1764752a73807376d44452e. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f797d4bad32 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250111 02:58:28.859439 21809 tablet_bootstrap.cc:492] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33: Bootstrap starting.
I20250111 02:58:28.872879 21809 tablet_bootstrap.cc:654] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:28.874233 21812 tablet_bootstrap.cc:492] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da: No bootstrap required, opened a new log
I20250111 02:58:28.874682 21812 ts_tablet_manager.cc:1397] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da: Time spent bootstrapping tablet: real 0.101s user 0.010s sys 0.001s
I20250111 02:58:28.875367 21820 ts_tablet_manager.cc:1918] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:28.875782 21820 log.cc:1198] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/e945d233bffc4251bcd74340c3e6f8c6
I20250111 02:58:28.876487 21820 ts_tablet_manager.cc:1939] T e945d233bffc4251bcd74340c3e6f8c6 P b6ee352ce89c48d38b467042c4c3f9fc: Deleting consensus metadata
I20250111 02:58:28.877277 21810 tablet_bootstrap.cc:492] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc: No bootstrap required, opened a new log
I20250111 02:58:28.877734 21810 ts_tablet_manager.cc:1397] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.001s
I20250111 02:58:28.879423 21020 catalog_manager.cc:4872] TS b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313): tablet e945d233bffc4251bcd74340c3e6f8c6 (table default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]) successfully deleted
I20250111 02:58:28.880264 21810 raft_consensus.cc:357] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.880970 21810 raft_consensus.cc:383] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:28.881248 21810 raft_consensus.cc:738] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Initialized, Role: FOLLOWER
I20250111 02:58:28.881886 21810 consensus_queue.cc:260] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.889103 21810 ts_tablet_manager.cc:1428] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20250111 02:58:28.890115 21810 tablet_bootstrap.cc:492] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc: Bootstrap starting.
I20250111 02:58:28.896171 21810 tablet_bootstrap.cc:654] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:28.901945 21812 raft_consensus.cc:357] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.902758 21812 raft_consensus.cc:383] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:28.903079 21812 raft_consensus.cc:738] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Initialized, Role: FOLLOWER
I20250111 02:58:28.907812 21812 consensus_queue.cc:260] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.906950 21821 ts_tablet_manager.cc:1918] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:28.909132 21821 log.cc:1198] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/e945d233bffc4251bcd74340c3e6f8c6
I20250111 02:58:28.910071 21821 ts_tablet_manager.cc:1939] T e945d233bffc4251bcd74340c3e6f8c6 P 2754981a32a44bf889c5d2a226e013da: Deleting consensus metadata
I20250111 02:58:28.910302 21819 ts_tablet_manager.cc:1918] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:28.913975 21824 raft_consensus.cc:491] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:28.914590 21824 raft_consensus.cc:513] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.920419 21812 ts_tablet_manager.cc:1428] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da: Time spent starting tablet: real 0.045s user 0.008s sys 0.000s
I20250111 02:58:28.923296 21824 leader_election.cc:290] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313), 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021)
I20250111 02:58:28.935479 21812 tablet_bootstrap.cc:492] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da: Bootstrap starting.
I20250111 02:58:28.937608 21380 tablet_service.cc:1514] Processing DeleteTablet for tablet 55be07345aff496c8cc910f921254ded with delete_type TABLET_DATA_DELETED (replaced by tablet caf9a22c859d46e29bb216e6248bfa1b) from {username='slave'} at 127.0.0.1:36052
I20250111 02:58:28.938079 21019 catalog_manager.cc:4872] TS 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): tablet e945d233bffc4251bcd74340c3e6f8c6 (table default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]) successfully deleted
W20250111 02:58:28.939849 21019 catalog_manager.cc:4839] TS 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): delete failed for tablet e945d233bffc4251bcd74340c3e6f8c6 because the tablet was not found. No further retry: Not found: Tablet not found: e945d233bffc4251bcd74340c3e6f8c6
I20250111 02:58:28.940747 21809 tablet_bootstrap.cc:492] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33: No bootstrap required, opened a new log
I20250111 02:58:28.940829 21821 tablet_replica.cc:331] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da: stopping tablet replica
I20250111 02:58:28.941581 21821 raft_consensus.cc:2238] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:28.942111 21821 raft_consensus.cc:2267] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Raft consensus is shut down!
@ 0x7f797d4bacea _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20250111 02:58:28.949694 21812 tablet_bootstrap.cc:654] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:28.941674 21809 ts_tablet_manager.cc:1397] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33: Time spent bootstrapping tablet: real 0.082s user 0.007s sys 0.006s
I20250111 02:58:28.910830 21819 log.cc:1198] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/55be07345aff496c8cc910f921254ded
I20250111 02:58:28.953277 21821 ts_tablet_manager.cc:1905] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:28.964530 21819 ts_tablet_manager.cc:1939] T 55be07345aff496c8cc910f921254ded P 6830bca1186e4084868808884170cc33: Deleting consensus metadata
W20250111 02:58:28.975798 21635 leader_election.cc:343] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313): Not found: Tablet not found: e945d233bffc4251bcd74340c3e6f8c6
I20250111 02:58:28.953819 21809 raft_consensus.cc:357] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.976562 21809 raft_consensus.cc:383] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:28.976925 21809 raft_consensus.cc:738] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830bca1186e4084868808884170cc33, State: Initialized, Role: FOLLOWER
I20250111 02:58:28.980918 21809 consensus_queue.cc:260] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:28.997915 21819 tablet_replica.cc:331] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33: stopping tablet replica
I20250111 02:58:28.999078 21809 ts_tablet_manager.cc:1428] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33: Time spent starting tablet: real 0.048s user 0.007s sys 0.001s
I20250111 02:58:29.000033 21809 tablet_bootstrap.cc:492] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33: Bootstrap starting.
I20250111 02:58:29.001823 21022 catalog_manager.cc:4872] TS 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): tablet 55be07345aff496c8cc910f921254ded (table default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]) successfully deleted
I20250111 02:58:29.003604 21819 raft_consensus.cc:2238] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:29.004977 21819 raft_consensus.cc:2267] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:29.007442 21819 ts_tablet_manager.cc:1905] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:29.013154 21029 master_service.cc:946] ReplaceTablet: received request to replace tablet dfbc1d2cb1764752a73807376d44452e from {username='slave'} at 127.0.0.1:49668
I20250111 02:58:29.012552 21810 tablet_bootstrap.cc:492] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc: No bootstrap required, opened a new log
I20250111 02:58:29.017702 21810 ts_tablet_manager.cc:1397] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc: Time spent bootstrapping tablet: real 0.128s user 0.005s sys 0.014s
W20250111 02:58:29.018633 21634 leader_election.cc:343] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): Not found: Tablet not found: e945d233bffc4251bcd74340c3e6f8c6
I20250111 02:58:29.019204 21634 leader_election.cc:304] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6830bca1186e4084868808884170cc33; no voters: 2754981a32a44bf889c5d2a226e013da, b6ee352ce89c48d38b467042c4c3f9fc
W20250111 02:58:29.020004 21634 raft_consensus.cc:2681] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250111 02:58:29.020417 21810 raft_consensus.cc:357] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.021191 21810 raft_consensus.cc:383] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.021497 21810 raft_consensus.cc:738] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.034904 21029 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Sending DeleteTablet for 3 replicas of tablet dfbc1d2cb1764752a73807376d44452e
I20250111 02:58:29.036520 21029 catalog_manager.cc:6596] ReplaceTablet: tablet dfbc1d2cb1764752a73807376d44452e deleted and replaced by tablet 2ba3ff8bb918447e83aba63d1357a26c
I20250111 02:58:29.037683 21679 tablet_service.cc:1514] Processing DeleteTablet for tablet dfbc1d2cb1764752a73807376d44452e with delete_type TABLET_DATA_DELETED (replaced by tablet 2ba3ff8bb918447e83aba63d1357a26c) from {username='slave'} at 127.0.0.1:44898
I20250111 02:58:29.038811 21536 tablet_service.cc:1514] Processing DeleteTablet for tablet dfbc1d2cb1764752a73807376d44452e with delete_type TABLET_DATA_DELETED (replaced by tablet 2ba3ff8bb918447e83aba63d1357a26c) from {username='slave'} at 127.0.0.1:52636
I20250111 02:58:29.040166 21029 master_service.cc:946] ReplaceTablet: received request to replace tablet 3ca2aca3ca0447d8af6941450a537e36 from {username='slave'} at 127.0.0.1:49668
W20250111 02:58:29.040561 21020 catalog_manager.cc:4852] TS b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313): delete failed for tablet dfbc1d2cb1764752a73807376d44452e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfbc1d2cb1764752a73807376d44452e already in progress: creating tablet
I20250111 02:58:29.042699 21380 tablet_service.cc:1514] Processing DeleteTablet for tablet dfbc1d2cb1764752a73807376d44452e with delete_type TABLET_DATA_DELETED (replaced by tablet 2ba3ff8bb918447e83aba63d1357a26c) from {username='slave'} at 127.0.0.1:36052
I20250111 02:58:29.045168 21812 tablet_bootstrap.cc:492] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da: No bootstrap required, opened a new log
I20250111 02:58:29.045703 21812 ts_tablet_manager.cc:1397] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da: Time spent bootstrapping tablet: real 0.110s user 0.009s sys 0.005s
I20250111 02:58:29.048542 21812 raft_consensus.cc:357] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.051288 21809 tablet_bootstrap.cc:654] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.051884 21810 consensus_queue.cc:260] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.052867 21812 raft_consensus.cc:383] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.053486 21812 raft_consensus.cc:738] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.061715 21810 ts_tablet_manager.cc:1428] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc: Time spent starting tablet: real 0.044s user 0.004s sys 0.002s
I20250111 02:58:29.065088 21821 ts_tablet_manager.cc:1918] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.062683 21812 consensus_queue.cc:260] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.073825 21679 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ca2aca3ca0447d8af6941450a537e36 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a6972eb15e94821ae39d46ca26fe940) from {username='slave'} at 127.0.0.1:44898
I20250111 02:58:29.075367 21029 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Sending DeleteTablet for 3 replicas of tablet 3ca2aca3ca0447d8af6941450a537e36
I20250111 02:58:29.077527 21380 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ca2aca3ca0447d8af6941450a537e36 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a6972eb15e94821ae39d46ca26fe940) from {username='slave'} at 127.0.0.1:36052
I20250111 02:58:29.079310 21679 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ca2aca3ca0447d8af6941450a537e36 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a6972eb15e94821ae39d46ca26fe940) from {username='slave'} at 127.0.0.1:44898
I20250111 02:58:29.079891 21029 catalog_manager.cc:6596] ReplaceTablet: tablet 3ca2aca3ca0447d8af6941450a537e36 deleted and replaced by tablet 1a6972eb15e94821ae39d46ca26fe940
I20250111 02:58:29.083367 21536 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ca2aca3ca0447d8af6941450a537e36 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a6972eb15e94821ae39d46ca26fe940) from {username='slave'} at 127.0.0.1:52636
I20250111 02:58:29.084164 21820 tablet_replica.cc:331] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc: stopping tablet replica
I20250111 02:58:29.084923 21820 raft_consensus.cc:2238] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:29.085409 21820 raft_consensus.cc:2267] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:29.088119 21820 ts_tablet_manager.cc:1905] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:29.096515 21812 ts_tablet_manager.cc:1428] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da: Time spent starting tablet: real 0.050s user 0.006s sys 0.000s
I20250111 02:58:29.096953 21821 log.cc:1198] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/55be07345aff496c8cc910f921254ded
I20250111 02:58:29.098078 21821 ts_tablet_manager.cc:1939] T 55be07345aff496c8cc910f921254ded P 2754981a32a44bf889c5d2a226e013da: Deleting consensus metadata
@ 0x7f797d4bac7a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20250111 02:58:29.106652 21536 tablet_service.cc:1514] Processing DeleteTablet for tablet dfbc1d2cb1764752a73807376d44452e with delete_type TABLET_DATA_DELETED (replaced by tablet 2ba3ff8bb918447e83aba63d1357a26c) from {username='slave'} at 127.0.0.1:52636
I20250111 02:58:29.113065 21812 tablet_bootstrap.cc:492] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da: Bootstrap starting.
I20250111 02:58:29.115716 21819 ts_tablet_manager.cc:1918] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.116210 21819 log.cc:1198] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/e945d233bffc4251bcd74340c3e6f8c6
I20250111 02:58:29.116936 21819 ts_tablet_manager.cc:1939] T e945d233bffc4251bcd74340c3e6f8c6 P 6830bca1186e4084868808884170cc33: Deleting consensus metadata
I20250111 02:58:29.119789 21019 catalog_manager.cc:4872] TS 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): tablet 55be07345aff496c8cc910f921254ded (table default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]) successfully deleted
I20250111 02:58:29.120306 21821 tablet_replica.cc:331] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da: stopping tablet replica
W20250111 02:58:29.120615 21019 catalog_manager.cc:4852] TS 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): delete failed for tablet dfbc1d2cb1764752a73807376d44452e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfbc1d2cb1764752a73807376d44452e already in progress: creating tablet
I20250111 02:58:29.121021 21821 raft_consensus.cc:2238] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:29.121500 21821 raft_consensus.cc:2267] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:29.126262 21809 tablet_bootstrap.cc:492] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33: No bootstrap required, opened a new log
I20250111 02:58:29.126843 21809 ts_tablet_manager.cc:1397] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33: Time spent bootstrapping tablet: real 0.127s user 0.009s sys 0.004s
I20250111 02:58:29.129890 21820 ts_tablet_manager.cc:1918] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.130287 21820 log.cc:1198] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/3ca2aca3ca0447d8af6941450a537e36
I20250111 02:58:29.130874 21380 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6972eb15e94821ae39d46ca26fe940 (DEFAULT_TABLE table=default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:29.132143 21380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a6972eb15e94821ae39d46ca26fe940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:29.132046 21536 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6972eb15e94821ae39d46ca26fe940 (DEFAULT_TABLE table=default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:29.133091 21536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a6972eb15e94821ae39d46ca26fe940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:29.137221 21822 raft_consensus.cc:491] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:29.137821 21822 raft_consensus.cc:513] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.148921 21822 leader_election.cc:290] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:29.149536 21822 raft_consensus.cc:491] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:29.149940 21822 raft_consensus.cc:513] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.152413 21381 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba3ff8bb918447e83aba63d1357a26c (DEFAULT_TABLE table=default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:29.153604 21679 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6972eb15e94821ae39d46ca26fe940 (DEFAULT_TABLE table=default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:29.154858 21679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a6972eb15e94821ae39d46ca26fe940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:29.155738 21822 leader_election.cc:290] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:29.159853 21535 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba3ff8bb918447e83aba63d1357a26c (DEFAULT_TABLE table=default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:29.161226 21535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ba3ff8bb918447e83aba63d1357a26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:29.163185 21680 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba3ff8bb918447e83aba63d1357a26c (DEFAULT_TABLE table=default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:29.164523 21680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ba3ff8bb918447e83aba63d1357a26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:29.169240 21810 tablet_bootstrap.cc:492] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc: Bootstrap starting.
I20250111 02:58:29.173226 21821 ts_tablet_manager.cc:1905] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:29.175737 21812 tablet_bootstrap.cc:654] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.175838 21810 tablet_bootstrap.cc:654] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.186177 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbc1d2cb1764752a73807376d44452e" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2754981a32a44bf889c5d2a226e013da" is_pre_election: true
I20250111 02:58:29.186965 21400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf9a22c859d46e29bb216e6248bfa1b" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2754981a32a44bf889c5d2a226e013da" is_pre_election: true
W20250111 02:58:29.188701 21489 leader_election.cc:343] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:58:29.189677 21489 leader_election.cc:343] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:58:29.190899 21810 tablet_bootstrap.cc:492] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc: No bootstrap required, opened a new log
I20250111 02:58:29.131008 21820 ts_tablet_manager.cc:1939] T 3ca2aca3ca0447d8af6941450a537e36 P b6ee352ce89c48d38b467042c4c3f9fc: Deleting consensus metadata
I20250111 02:58:29.191375 21810 ts_tablet_manager.cc:1397] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.008s
I20250111 02:58:29.193871 21020 catalog_manager.cc:4872] TS b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313): tablet 3ca2aca3ca0447d8af6941450a537e36 (table default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]) successfully deleted
I20250111 02:58:29.195014 21820 tablet_replica.cc:331] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc: stopping tablet replica
I20250111 02:58:29.195855 21820 raft_consensus.cc:2238] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:29.199791 21809 raft_consensus.cc:357] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.200625 21809 raft_consensus.cc:383] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.200925 21809 raft_consensus.cc:738] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830bca1186e4084868808884170cc33, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.203579 21820 raft_consensus.cc:2267] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:29.206710 21812 tablet_bootstrap.cc:492] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da: No bootstrap required, opened a new log
I20250111 02:58:29.207253 21812 ts_tablet_manager.cc:1397] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da: Time spent bootstrapping tablet: real 0.100s user 0.012s sys 0.000s
I20250111 02:58:29.207547 21381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ba3ff8bb918447e83aba63d1357a26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:29.198905 21810 raft_consensus.cc:357] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.210951 21810 raft_consensus.cc:383] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.211380 21810 raft_consensus.cc:738] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.212579 21810 consensus_queue.cc:260] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.224548 21820 ts_tablet_manager.cc:1905] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:29.210245 21812 raft_consensus.cc:357] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.228775 21812 raft_consensus.cc:383] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.229111 21812 raft_consensus.cc:738] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.229889 21810 ts_tablet_manager.cc:1428] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc: Time spent starting tablet: real 0.038s user 0.007s sys 0.002s
I20250111 02:58:29.229816 21812 consensus_queue.cc:260] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.231529 21810 tablet_bootstrap.cc:492] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc: Bootstrap starting.
I20250111 02:58:29.243522 21809 consensus_queue.cc:260] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.250326 21026 master_service.cc:946] ReplaceTablet: received request to replace tablet dfbc1d2cb1764752a73807376d44452e from {username='slave'} at 127.0.0.1:49668
I20250111 02:58:29.252522 21810 tablet_bootstrap.cc:654] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.263590 21812 ts_tablet_manager.cc:1428] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da: Time spent starting tablet: real 0.056s user 0.007s sys 0.001s
I20250111 02:58:29.265640 21810 tablet_bootstrap.cc:492] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc: No bootstrap required, opened a new log
I20250111 02:58:29.266161 21810 ts_tablet_manager.cc:1397] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.007s
I20250111 02:58:29.267627 21022 catalog_manager.cc:4872] TS 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): tablet e945d233bffc4251bcd74340c3e6f8c6 (table default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]) successfully deleted
W20250111 02:58:29.269678 21022 catalog_manager.cc:4852] TS 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): delete failed for tablet dfbc1d2cb1764752a73807376d44452e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfbc1d2cb1764752a73807376d44452e already in progress: creating tablet
I20250111 02:58:29.270539 21810 raft_consensus.cc:357] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.271340 21810 raft_consensus.cc:383] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.274533 21812 tablet_bootstrap.cc:492] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da: Bootstrap starting.
I20250111 02:58:29.274907 21810 raft_consensus.cc:738] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.275801 21810 consensus_queue.cc:260] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.280925 21810 ts_tablet_manager.cc:1428] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250111 02:58:29.285408 21809 ts_tablet_manager.cc:1428] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33: Time spent starting tablet: real 0.158s user 0.005s sys 0.001s
I20250111 02:58:29.290475 21026 master_service.cc:946] ReplaceTablet: received request to replace tablet caf9a22c859d46e29bb216e6248bfa1b from {username='slave'} at 127.0.0.1:49668
I20250111 02:58:29.293514 21809 tablet_bootstrap.cc:492] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33: Bootstrap starting.
I20250111 02:58:29.297906 21819 tablet_replica.cc:331] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33: stopping tablet replica
I20250111 02:58:29.298738 21819 raft_consensus.cc:2238] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:29.307611 21820 ts_tablet_manager.cc:1918] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.308027 21820 log.cc:1198] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/dfbc1d2cb1764752a73807376d44452e
I20250111 02:58:29.307360 21819 raft_consensus.cc:2267] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:29.308723 21820 ts_tablet_manager.cc:1939] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc: Deleting consensus metadata
I20250111 02:58:29.310799 21020 catalog_manager.cc:4872] TS b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313): tablet dfbc1d2cb1764752a73807376d44452e (table default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]) successfully deleted
I20250111 02:58:29.314885 21812 tablet_bootstrap.cc:654] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.318603 21844 raft_consensus.cc:491] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:29.319133 21844 raft_consensus.cc:513] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.320212 21026 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Sending DeleteTablet for 3 replicas of tablet caf9a22c859d46e29bb216e6248bfa1b
I20250111 02:58:29.321275 21844 leader_election.cc:290] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:29.322134 21379 tablet_service.cc:1514] Processing DeleteTablet for tablet caf9a22c859d46e29bb216e6248bfa1b with delete_type TABLET_DATA_DELETED (replaced by tablet 9c63181792b840bca29360a71def7f1c) from {username='slave'} at 127.0.0.1:36052
I20250111 02:58:29.322743 21400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba3ff8bb918447e83aba63d1357a26c" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2754981a32a44bf889c5d2a226e013da" is_pre_election: true
W20250111 02:58:29.325148 21489 leader_election.cc:343] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:58:29.326848 21535 tablet_service.cc:1514] Processing DeleteTablet for tablet caf9a22c859d46e29bb216e6248bfa1b with delete_type TABLET_DATA_DELETED (replaced by tablet 9c63181792b840bca29360a71def7f1c) from {username='slave'} at 127.0.0.1:52636
I20250111 02:58:29.327605 21026 catalog_manager.cc:6596] ReplaceTablet: tablet caf9a22c859d46e29bb216e6248bfa1b deleted and replaced by tablet 9c63181792b840bca29360a71def7f1c
I20250111 02:58:29.335923 21821 ts_tablet_manager.cc:1918] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.336441 21821 log.cc:1198] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/3ca2aca3ca0447d8af6941450a537e36
I20250111 02:58:29.337332 21821 ts_tablet_manager.cc:1939] T 3ca2aca3ca0447d8af6941450a537e36 P 2754981a32a44bf889c5d2a226e013da: Deleting consensus metadata
I20250111 02:58:29.337543 21820 tablet_replica.cc:331] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc: stopping tablet replica
I20250111 02:58:29.338467 21820 raft_consensus.cc:2238] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:29.339097 21820 raft_consensus.cc:2267] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:29.343043 21681 tablet_service.cc:1514] Processing DeleteTablet for tablet caf9a22c859d46e29bb216e6248bfa1b with delete_type TABLET_DATA_DELETED (replaced by tablet 9c63181792b840bca29360a71def7f1c) from {username='slave'} at 127.0.0.1:44898
I20250111 02:58:29.347859 21820 ts_tablet_manager.cc:1905] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:29.348044 21019 catalog_manager.cc:4872] TS 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): tablet 3ca2aca3ca0447d8af6941450a537e36 (table default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]) successfully deleted
I20250111 02:58:29.349356 21678 tablet_service.cc:1514] Processing DeleteTablet for tablet caf9a22c859d46e29bb216e6248bfa1b with delete_type TABLET_DATA_DELETED (replaced by tablet 9c63181792b840bca29360a71def7f1c) from {username='slave'} at 127.0.0.1:44898
I20250111 02:58:29.354473 21535 tablet_service.cc:1467] Processing CreateTablet for tablet 9c63181792b840bca29360a71def7f1c (DEFAULT_TABLE table=default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:29.355942 21535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c63181792b840bca29360a71def7f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:29.361624 21678 tablet_service.cc:1467] Processing CreateTablet for tablet 9c63181792b840bca29360a71def7f1c (DEFAULT_TABLE table=default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:29.363540 21678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c63181792b840bca29360a71def7f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:29.369153 21819 ts_tablet_manager.cc:1905] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:29.379779 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbc1d2cb1764752a73807376d44452e" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33" is_pre_election: true
I20250111 02:58:29.385295 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba3ff8bb918447e83aba63d1357a26c" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33" is_pre_election: true
W20250111 02:58:29.387003 21492 leader_election.cc:343] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:58:29.387570 21492 leader_election.cc:304] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b6ee352ce89c48d38b467042c4c3f9fc; no voters: 2754981a32a44bf889c5d2a226e013da, 6830bca1186e4084868808884170cc33
I20250111 02:58:29.387856 21829 raft_consensus.cc:491] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250111 02:58:29.388612 21492 raft_consensus.cc:2681] T dfbc1d2cb1764752a73807376d44452e P b6ee352ce89c48d38b467042c4c3f9fc: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250111 02:58:29.388617 21829 raft_consensus.cc:513] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.392477 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf9a22c859d46e29bb216e6248bfa1b" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33" is_pre_election: true
I20250111 02:58:29.393268 21701 raft_consensus.cc:2463] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6ee352ce89c48d38b467042c4c3f9fc in term 0.
W20250111 02:58:29.394373 21492 leader_election.cc:343] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:58:29.394870 21492 leader_election.cc:304] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b6ee352ce89c48d38b467042c4c3f9fc; no voters: 2754981a32a44bf889c5d2a226e013da, 6830bca1186e4084868808884170cc33
I20250111 02:58:29.395817 21844 raft_consensus.cc:2743] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:58:29.396459 21829 leader_election.cc:290] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:29.403586 21492 leader_election.cc:304] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6830bca1186e4084868808884170cc33, b6ee352ce89c48d38b467042c4c3f9fc; no voters: 2754981a32a44bf889c5d2a226e013da
W20250111 02:58:29.404543 21492 raft_consensus.cc:2681] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250111 02:58:29.409463 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf9a22c859d46e29bb216e6248bfa1b" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" is_pre_election: true
W20250111 02:58:29.411372 21335 leader_election.cc:343] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313): Illegal state: Tablet not RUNNING: STOPPED
I20250111 02:58:29.413558 21381 tablet_service.cc:1467] Processing CreateTablet for tablet 9c63181792b840bca29360a71def7f1c (DEFAULT_TABLE table=default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:29.415217 21381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c63181792b840bca29360a71def7f1c. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f797d4bac42 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20250111 02:58:29.417379 21821 tablet_replica.cc:331] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da: stopping tablet replica
I20250111 02:58:29.423908 21821 raft_consensus.cc:2238] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:29.424607 21821 raft_consensus.cc:2267] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:29.432163 21810 tablet_bootstrap.cc:492] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc: Bootstrap starting.
I20250111 02:58:29.434085 21821 ts_tablet_manager.cc:1905] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:29.435396 21809 tablet_bootstrap.cc:654] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.443150 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf9a22c859d46e29bb216e6248bfa1b" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33" is_pre_election: true
I20250111 02:58:29.443830 21701 raft_consensus.cc:2463] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2754981a32a44bf889c5d2a226e013da in term 0.
I20250111 02:58:29.445314 21337 leader_election.cc:304] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2754981a32a44bf889c5d2a226e013da, 6830bca1186e4084868808884170cc33; no voters: b6ee352ce89c48d38b467042c4c3f9fc
I20250111 02:58:29.446153 21810 tablet_bootstrap.cc:654] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc: Neither blocks nor log segments found. Creating new log.
W20250111 02:58:29.446262 21337 raft_consensus.cc:2681] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250111 02:58:29.454710 21810 tablet_bootstrap.cc:492] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc: No bootstrap required, opened a new log
I20250111 02:58:29.455235 21810 ts_tablet_manager.cc:1397] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.006s
I20250111 02:58:29.461027 21812 tablet_bootstrap.cc:492] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da: No bootstrap required, opened a new log
I20250111 02:58:29.461539 21812 ts_tablet_manager.cc:1397] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da: Time spent bootstrapping tablet: real 0.187s user 0.020s sys 0.034s
I20250111 02:58:29.464730 21026 master_service.cc:946] ReplaceTablet: received request to replace tablet 1a6972eb15e94821ae39d46ca26fe940 from {username='slave'} at 127.0.0.1:49668
I20250111 02:58:29.474264 21819 ts_tablet_manager.cc:1918] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.474702 21819 log.cc:1198] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/3ca2aca3ca0447d8af6941450a537e36
I20250111 02:58:29.476148 21810 raft_consensus.cc:357] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.476985 21810 raft_consensus.cc:383] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.477275 21810 raft_consensus.cc:738] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.477991 21810 consensus_queue.cc:260] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.480185 21809 tablet_bootstrap.cc:492] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33: No bootstrap required, opened a new log
I20250111 02:58:29.480580 21810 ts_tablet_manager.cc:1428] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20250111 02:58:29.480666 21809 ts_tablet_manager.cc:1397] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33: Time spent bootstrapping tablet: real 0.187s user 0.014s sys 0.000s
I20250111 02:58:29.484184 21812 raft_consensus.cc:357] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.485016 21812 raft_consensus.cc:383] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.485332 21812 raft_consensus.cc:738] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.486037 21812 consensus_queue.cc:260] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.490725 21820 ts_tablet_manager.cc:1918] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.490780 21026 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Sending DeleteTablet for 3 replicas of tablet 1a6972eb15e94821ae39d46ca26fe940
I20250111 02:58:29.491202 21820 log.cc:1198] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/caf9a22c859d46e29bb216e6248bfa1b
I20250111 02:58:29.490778 21809 raft_consensus.cc:357] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.491573 21809 raft_consensus.cc:383] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.491879 21809 raft_consensus.cc:738] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830bca1186e4084868808884170cc33, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.491932 21820 ts_tablet_manager.cc:1939] T caf9a22c859d46e29bb216e6248bfa1b P b6ee352ce89c48d38b467042c4c3f9fc: Deleting consensus metadata
I20250111 02:58:29.492391 21026 catalog_manager.cc:6596] ReplaceTablet: tablet 1a6972eb15e94821ae39d46ca26fe940 deleted and replaced by tablet 1ddafc3983b543978beb74d38cf9936e
I20250111 02:58:29.493875 21812 ts_tablet_manager.cc:1428] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da: Time spent starting tablet: real 0.032s user 0.007s sys 0.001s
I20250111 02:58:29.495007 21812 tablet_bootstrap.cc:492] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da: Bootstrap starting.
I20250111 02:58:29.495664 21819 ts_tablet_manager.cc:1939] T 3ca2aca3ca0447d8af6941450a537e36 P 6830bca1186e4084868808884170cc33: Deleting consensus metadata
I20250111 02:58:29.498821 21819 tablet_replica.cc:331] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33: stopping tablet replica
I20250111 02:58:29.499539 21819 raft_consensus.cc:2238] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:29.500065 21819 raft_consensus.cc:2267] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:29.500882 21380 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a6972eb15e94821ae39d46ca26fe940 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ddafc3983b543978beb74d38cf9936e) from {username='slave'} at 127.0.0.1:36052
I20250111 02:58:29.502328 21535 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a6972eb15e94821ae39d46ca26fe940 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ddafc3983b543978beb74d38cf9936e) from {username='slave'} at 127.0.0.1:52636
I20250111 02:58:29.502804 21022 catalog_manager.cc:4872] TS 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): tablet 3ca2aca3ca0447d8af6941450a537e36 (table default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]) successfully deleted
I20250111 02:58:29.500824 21809 consensus_queue.cc:260] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.508345 21809 ts_tablet_manager.cc:1428] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20250111 02:58:29.509410 21809 tablet_bootstrap.cc:492] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33: Bootstrap starting.
I20250111 02:58:29.521220 21812 tablet_bootstrap.cc:654] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.525835 21820 tablet_replica.cc:331] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc: stopping tablet replica
I20250111 02:58:29.526623 21820 raft_consensus.cc:2238] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:29.526813 21819 ts_tablet_manager.cc:1905] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:29.527123 21820 raft_consensus.cc:2267] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:29.530777 21020 catalog_manager.cc:4872] TS b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313): tablet caf9a22c859d46e29bb216e6248bfa1b (table default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]) successfully deleted
W20250111 02:58:29.535624 21022 catalog_manager.cc:4839] TS 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): delete failed for tablet 3ca2aca3ca0447d8af6941450a537e36 because the tablet was not found. No further retry: Not found: Tablet not found: 3ca2aca3ca0447d8af6941450a537e36
I20250111 02:58:29.544452 21809 tablet_bootstrap.cc:654] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33: Neither blocks nor log segments found. Creating new log.
@ 0x7f797d4b9f3e std::__1::__function::__func<>::operator()() at ??:0
@ 0x59ea05 std::__1::__function::__value_func<>::operator()() at ??:?
I20250111 02:58:29.547912 21381 tablet_service.cc:1514] Processing DeleteTablet for tablet dfbc1d2cb1764752a73807376d44452e with delete_type TABLET_DATA_DELETED (replaced by tablet 2ba3ff8bb918447e83aba63d1357a26c) from {username='slave'} at 127.0.0.1:36052
I20250111 02:58:29.549469 21820 ts_tablet_manager.cc:1905] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x59e83a std::__1::function<>::operator()() at ??:?
I20250111 02:58:29.551764 21821 ts_tablet_manager.cc:1918] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.552206 21821 log.cc:1198] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/caf9a22c859d46e29bb216e6248bfa1b
I20250111 02:58:29.552913 21821 ts_tablet_manager.cc:1939] T caf9a22c859d46e29bb216e6248bfa1b P 2754981a32a44bf889c5d2a226e013da: Deleting consensus metadata
I20250111 02:58:29.555635 21019 catalog_manager.cc:4872] TS 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): tablet caf9a22c859d46e29bb216e6248bfa1b (table default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]) successfully deleted
I20250111 02:58:29.558574 21679 tablet_service.cc:1514] Processing DeleteTablet for tablet dfbc1d2cb1764752a73807376d44452e with delete_type TABLET_DATA_DELETED (replaced by tablet 2ba3ff8bb918447e83aba63d1357a26c) from {username='slave'} at 127.0.0.1:44898
W20250111 02:58:29.566524 21019 catalog_manager.cc:4852] TS 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): delete failed for tablet 1a6972eb15e94821ae39d46ca26fe940 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a6972eb15e94821ae39d46ca26fe940 already in progress: creating tablet
I20250111 02:58:29.567636 21821 tablet_replica.cc:331] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da: stopping tablet replica
I20250111 02:58:29.568471 21821 raft_consensus.cc:2238] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:29.568995 21821 raft_consensus.cc:2267] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:29.570648 21029 master_service.cc:946] ReplaceTablet: received request to replace tablet caf9a22c859d46e29bb216e6248bfa1b from {username='slave'} at 127.0.0.1:49668
I20250111 02:58:29.573812 21809 tablet_bootstrap.cc:492] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33: No bootstrap required, opened a new log
I20250111 02:58:29.574270 21809 ts_tablet_manager.cc:1397] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33: Time spent bootstrapping tablet: real 0.065s user 0.010s sys 0.003s
I20250111 02:58:29.576716 21820 ts_tablet_manager.cc:1918] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.577087 21820 log.cc:1198] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/1a6972eb15e94821ae39d46ca26fe940
I20250111 02:58:29.577742 21820 ts_tablet_manager.cc:1939] T 1a6972eb15e94821ae39d46ca26fe940 P b6ee352ce89c48d38b467042c4c3f9fc: Deleting consensus metadata
I20250111 02:58:29.579512 21678 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a6972eb15e94821ae39d46ca26fe940 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ddafc3983b543978beb74d38cf9936e) from {username='slave'} at 127.0.0.1:44898
I20250111 02:58:29.586405 21821 ts_tablet_manager.cc:1905] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:29.593866 21809 raft_consensus.cc:357] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.594714 21809 raft_consensus.cc:383] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.595011 21809 raft_consensus.cc:738] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830bca1186e4084868808884170cc33, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.597734 21381 tablet_service.cc:1467] Processing CreateTablet for tablet 1ddafc3983b543978beb74d38cf9936e (DEFAULT_TABLE table=default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:29.598075 21819 ts_tablet_manager.cc:1918] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.598474 21819 log.cc:1198] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/caf9a22c859d46e29bb216e6248bfa1b
I20250111 02:58:29.598999 21381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ddafc3983b543978beb74d38cf9936e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:29.599201 21819 ts_tablet_manager.cc:1939] T caf9a22c859d46e29bb216e6248bfa1b P 6830bca1186e4084868808884170cc33: Deleting consensus metadata
I20250111 02:58:29.601938 21022 catalog_manager.cc:4872] TS 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): tablet caf9a22c859d46e29bb216e6248bfa1b (table default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]) successfully deleted
I20250111 02:58:29.603500 21809 consensus_queue.cc:260] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.605929 21809 ts_tablet_manager.cc:1428] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33: Time spent starting tablet: real 0.031s user 0.004s sys 0.002s
W20250111 02:58:29.612736 21022 catalog_manager.cc:4839] TS 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): delete failed for tablet caf9a22c859d46e29bb216e6248bfa1b because the tablet was not found. No further retry: Not found: Tablet not found: caf9a22c859d46e29bb216e6248bfa1b
I20250111 02:58:29.612864 21020 catalog_manager.cc:4872] TS b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313): tablet 1a6972eb15e94821ae39d46ca26fe940 (table default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]) successfully deleted
I20250111 02:58:29.606900 21809 tablet_bootstrap.cc:492] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33: Bootstrap starting.
I20250111 02:58:29.619903 21819 tablet_replica.cc:331] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33: stopping tablet replica
I20250111 02:58:29.620682 21819 raft_consensus.cc:2238] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:29.621204 21819 raft_consensus.cc:2267] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:29.625772 21819 ts_tablet_manager.cc:1905] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:29.636273 21809 tablet_bootstrap.cc:654] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.640566 21535 tablet_service.cc:1467] Processing CreateTablet for tablet 1ddafc3983b543978beb74d38cf9936e (DEFAULT_TABLE table=default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:29.641904 21535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ddafc3983b543978beb74d38cf9936e. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f79748018d7 kudu::Thread::SuperviseThread() at ??:0
@ 0x468a00 __tsan_thread_start_func at ??:?
I20250111 02:58:29.650704 21809 tablet_bootstrap.cc:492] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33: No bootstrap required, opened a new log
I20250111 02:58:29.653719 21809 ts_tablet_manager.cc:1397] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.000s
I20250111 02:58:29.653937 21678 tablet_service.cc:1467] Processing CreateTablet for tablet 1ddafc3983b543978beb74d38cf9936e (DEFAULT_TABLE table=default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:29.655252 21678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ddafc3983b543978beb74d38cf9936e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:29.656955 21809 raft_consensus.cc:357] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.657655 21809 raft_consensus.cc:383] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.663499 21809 raft_consensus.cc:738] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830bca1186e4084868808884170cc33, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.668176 21809 consensus_queue.cc:260] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.672756 21809 ts_tablet_manager.cc:1428] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33: Time spent starting tablet: real 0.019s user 0.002s sys 0.003s
I20250111 02:58:29.673628 21809 tablet_bootstrap.cc:492] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33: Bootstrap starting.
I20250111 02:58:29.678928 21819 ts_tablet_manager.cc:1918] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.679373 21819 log.cc:1198] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/dfbc1d2cb1764752a73807376d44452e
I20250111 02:58:29.680027 21819 ts_tablet_manager.cc:1939] T dfbc1d2cb1764752a73807376d44452e P 6830bca1186e4084868808884170cc33: Deleting consensus metadata
I20250111 02:58:29.683107 21022 catalog_manager.cc:4872] TS 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): tablet dfbc1d2cb1764752a73807376d44452e (table default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]) successfully deleted
I20250111 02:58:29.686348 21812 tablet_bootstrap.cc:492] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da: No bootstrap required, opened a new log
I20250111 02:58:29.686810 21812 ts_tablet_manager.cc:1397] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da: Time spent bootstrapping tablet: real 0.192s user 0.011s sys 0.033s
I20250111 02:58:29.689663 21812 raft_consensus.cc:357] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.690426 21812 raft_consensus.cc:383] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.690702 21812 raft_consensus.cc:738] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.691550 21812 consensus_queue.cc:260] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.697317 21812 ts_tablet_manager.cc:1428] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250111 02:58:29.698414 21812 tablet_bootstrap.cc:492] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da: Bootstrap starting.
I20250111 02:58:29.699514 21380 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a6972eb15e94821ae39d46ca26fe940 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ddafc3983b543978beb74d38cf9936e) from {username='slave'} at 127.0.0.1:36052
I20250111 02:58:29.704449 21809 tablet_bootstrap.cc:654] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.712005 21819 tablet_replica.cc:331] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33: stopping tablet replica
I20250111 02:58:29.713816 21819 raft_consensus.cc:2238] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:29.714334 21812 tablet_bootstrap.cc:654] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.714376 21819 raft_consensus.cc:2267] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:29.716692 21819 ts_tablet_manager.cc:1905] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:29.717355 21028 master_service.cc:946] ReplaceTablet: received request to replace tablet dfbc1d2cb1764752a73807376d44452e from {username='slave'} at 127.0.0.1:49668
I20250111 02:58:29.724967 21809 tablet_bootstrap.cc:492] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33: No bootstrap required, opened a new log
I20250111 02:58:29.725414 21809 ts_tablet_manager.cc:1397] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.004s
I20250111 02:58:29.727874 21809 raft_consensus.cc:357] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.728624 21809 raft_consensus.cc:383] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.728873 21809 raft_consensus.cc:738] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830bca1186e4084868808884170cc33, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.731997 21809 consensus_queue.cc:260] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.743450 21809 ts_tablet_manager.cc:1428] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33: Time spent starting tablet: real 0.018s user 0.007s sys 0.001s
@ 0x7f7975e806db start_thread at ??:0
I20250111 02:58:29.758334 21812 tablet_bootstrap.cc:492] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da: No bootstrap required, opened a new log
I20250111 02:58:29.758749 21812 ts_tablet_manager.cc:1397] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da: Time spent bootstrapping tablet: real 0.061s user 0.012s sys 0.000s
I20250111 02:58:29.761171 21812 raft_consensus.cc:357] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.761878 21812 raft_consensus.cc:383] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.762303 21812 raft_consensus.cc:738] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.762931 21812 consensus_queue.cc:260] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.768544 21812 ts_tablet_manager.cc:1428] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250111 02:58:29.769455 21812 tablet_bootstrap.cc:492] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da: Bootstrap starting.
I20250111 02:58:29.770999 21810 tablet_bootstrap.cc:492] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc: Bootstrap starting.
I20250111 02:58:29.775224 21812 tablet_bootstrap.cc:654] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.767781 21819 ts_tablet_manager.cc:1918] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.776823 21810 tablet_bootstrap.cc:654] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.776894 21819 log.cc:1198] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/1a6972eb15e94821ae39d46ca26fe940
I20250111 02:58:29.777603 21819 ts_tablet_manager.cc:1939] T 1a6972eb15e94821ae39d46ca26fe940 P 6830bca1186e4084868808884170cc33: Deleting consensus metadata
I20250111 02:58:29.781450 21022 catalog_manager.cc:4872] TS 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): tablet 1a6972eb15e94821ae39d46ca26fe940 (table default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]) successfully deleted
I20250111 02:58:29.783573 21821 ts_tablet_manager.cc:1918] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.784094 21821 log.cc:1198] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/dfbc1d2cb1764752a73807376d44452e
I20250111 02:58:29.791366 21809 tablet_bootstrap.cc:492] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33: Bootstrap starting.
I20250111 02:58:29.796391 21821 ts_tablet_manager.cc:1939] T dfbc1d2cb1764752a73807376d44452e P 2754981a32a44bf889c5d2a226e013da: Deleting consensus metadata
I20250111 02:58:29.797963 21809 tablet_bootstrap.cc:654] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.799019 21028 master_service.cc:946] ReplaceTablet: received request to replace tablet 2ba3ff8bb918447e83aba63d1357a26c from {username='slave'} at 127.0.0.1:49668
I20250111 02:58:29.799420 21019 catalog_manager.cc:4872] TS 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): tablet dfbc1d2cb1764752a73807376d44452e (table default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]) successfully deleted
I20250111 02:58:29.800422 21821 tablet_replica.cc:331] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da: stopping tablet replica
I20250111 02:58:29.802335 21821 raft_consensus.cc:2238] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:29.802834 21821 raft_consensus.cc:2267] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:29.803040 21812 tablet_bootstrap.cc:492] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da: No bootstrap required, opened a new log
I20250111 02:58:29.803483 21812 ts_tablet_manager.cc:1397] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.007s
I20250111 02:58:29.805208 21821 ts_tablet_manager.cc:1905] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:29.805884 21812 raft_consensus.cc:357] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.806608 21812 raft_consensus.cc:383] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.809923 21809 tablet_bootstrap.cc:492] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33: No bootstrap required, opened a new log
I20250111 02:58:29.810356 21809 ts_tablet_manager.cc:1397] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.003s
I20250111 02:58:29.806859 21812 raft_consensus.cc:738] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.814168 21812 consensus_queue.cc:260] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.812860 21809 raft_consensus.cc:357] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.815235 21809 raft_consensus.cc:383] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.815518 21809 raft_consensus.cc:738] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830bca1186e4084868808884170cc33, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.817653 21810 tablet_bootstrap.cc:492] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc: No bootstrap required, opened a new log
I20250111 02:58:29.818059 21810 ts_tablet_manager.cc:1397] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.000s
I20250111 02:58:29.820381 21809 consensus_queue.cc:260] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.821496 21812 ts_tablet_manager.cc:1428] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20250111 02:58:29.821400 21821 ts_tablet_manager.cc:1918] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.821867 21821 log.cc:1198] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/1a6972eb15e94821ae39d46ca26fe940
I20250111 02:58:29.821684 21810 raft_consensus.cc:357] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.822383 21812 tablet_bootstrap.cc:492] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da: Bootstrap starting.
I20250111 02:58:29.822376 21810 raft_consensus.cc:383] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.822559 21821 ts_tablet_manager.cc:1939] T 1a6972eb15e94821ae39d46ca26fe940 P 2754981a32a44bf889c5d2a226e013da: Deleting consensus metadata
I20250111 02:58:29.822641 21810 raft_consensus.cc:738] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.826612 21809 ts_tablet_manager.cc:1428] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20250111 02:58:29.827849 21810 consensus_queue.cc:260] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.829234 21019 catalog_manager.cc:4872] TS 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): tablet 1a6972eb15e94821ae39d46ca26fe940 (table default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]) successfully deleted
I20250111 02:58:29.834864 21812 tablet_bootstrap.cc:654] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.848745 21810 ts_tablet_manager.cc:1428] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc: Time spent starting tablet: real 0.029s user 0.003s sys 0.002s
I20250111 02:58:29.861037 21028 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Sending DeleteTablet for 3 replicas of tablet 2ba3ff8bb918447e83aba63d1357a26c
I20250111 02:58:29.863144 21535 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba3ff8bb918447e83aba63d1357a26c with delete_type TABLET_DATA_DELETED (replaced by tablet b8cb51676b7346df96abe11be6d053cb) from {username='slave'} at 127.0.0.1:52636
I20250111 02:58:29.863875 21820 tablet_replica.cc:331] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc: stopping tablet replica
I20250111 02:58:29.864683 21820 raft_consensus.cc:2238] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:29.865170 21820 raft_consensus.cc:2267] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:29.866963 21820 ts_tablet_manager.cc:1905] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:29.879143 21678 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba3ff8bb918447e83aba63d1357a26c with delete_type TABLET_DATA_DELETED (replaced by tablet b8cb51676b7346df96abe11be6d053cb) from {username='slave'} at 127.0.0.1:44898
I20250111 02:58:29.879963 21819 tablet_replica.cc:331] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33: stopping tablet replica
I20250111 02:58:29.880611 21028 catalog_manager.cc:6596] ReplaceTablet: tablet 2ba3ff8bb918447e83aba63d1357a26c deleted and replaced by tablet b8cb51676b7346df96abe11be6d053cb
I20250111 02:58:29.880784 21819 raft_consensus.cc:2238] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:29.881283 21819 raft_consensus.cc:2267] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:29.884400 21819 ts_tablet_manager.cc:1905] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:29.892836 21678 tablet_service.cc:1467] Processing CreateTablet for tablet b8cb51676b7346df96abe11be6d053cb (DEFAULT_TABLE table=default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:29.893296 21535 tablet_service.cc:1467] Processing CreateTablet for tablet b8cb51676b7346df96abe11be6d053cb (DEFAULT_TABLE table=default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:29.893990 21678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8cb51676b7346df96abe11be6d053cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:29.894399 21535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8cb51676b7346df96abe11be6d053cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:29.903693 21812 tablet_bootstrap.cc:492] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da: No bootstrap required, opened a new log
I20250111 02:58:29.904148 21812 ts_tablet_manager.cc:1397] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da: Time spent bootstrapping tablet: real 0.082s user 0.019s sys 0.013s
I20250111 02:58:29.906718 21812 raft_consensus.cc:357] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.907429 21812 raft_consensus.cc:383] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.907686 21812 raft_consensus.cc:738] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.908279 21812 consensus_queue.cc:260] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.910526 21812 ts_tablet_manager.cc:1428] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20250111 02:58:29.941829 21819 ts_tablet_manager.cc:1918] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.942250 21819 log.cc:1198] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/2ba3ff8bb918447e83aba63d1357a26c
@ 0x7f797126171f clone at ??:0
I20250111 02:58:29.947278 21809 tablet_bootstrap.cc:492] T b8cb51676b7346df96abe11be6d053cb P 6830bca1186e4084868808884170cc33: Bootstrap starting.
I20250111 02:58:29.950516 21381 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba3ff8bb918447e83aba63d1357a26c with delete_type TABLET_DATA_DELETED (replaced by tablet b8cb51676b7346df96abe11be6d053cb) from {username='slave'} at 127.0.0.1:36052
I20250111 02:58:29.953299 21809 tablet_bootstrap.cc:654] T b8cb51676b7346df96abe11be6d053cb P 6830bca1186e4084868808884170cc33: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.953276 21380 tablet_service.cc:1467] Processing CreateTablet for tablet b8cb51676b7346df96abe11be6d053cb (DEFAULT_TABLE table=default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:29.954825 21380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8cb51676b7346df96abe11be6d053cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:29.957742 21819 ts_tablet_manager.cc:1939] T 2ba3ff8bb918447e83aba63d1357a26c P 6830bca1186e4084868808884170cc33: Deleting consensus metadata
I20250111 02:58:29.958001 21810 tablet_bootstrap.cc:492] T b8cb51676b7346df96abe11be6d053cb P b6ee352ce89c48d38b467042c4c3f9fc: Bootstrap starting.
I20250111 02:58:29.961575 21820 ts_tablet_manager.cc:1918] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:29.962023 21820 log.cc:1198] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/2ba3ff8bb918447e83aba63d1357a26c
I20250111 02:58:29.962693 21820 ts_tablet_manager.cc:1939] T 2ba3ff8bb918447e83aba63d1357a26c P b6ee352ce89c48d38b467042c4c3f9fc: Deleting consensus metadata
I20250111 02:58:29.963404 21810 tablet_bootstrap.cc:654] T b8cb51676b7346df96abe11be6d053cb P b6ee352ce89c48d38b467042c4c3f9fc: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:29.970571 21810 tablet_bootstrap.cc:492] T b8cb51676b7346df96abe11be6d053cb P b6ee352ce89c48d38b467042c4c3f9fc: No bootstrap required, opened a new log
I20250111 02:58:29.970959 21810 ts_tablet_manager.cc:1397] T b8cb51676b7346df96abe11be6d053cb P b6ee352ce89c48d38b467042c4c3f9fc: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20250111 02:58:29.973768 21810 raft_consensus.cc:357] T b8cb51676b7346df96abe11be6d053cb P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.974431 21810 raft_consensus.cc:383] T b8cb51676b7346df96abe11be6d053cb P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.974661 21810 raft_consensus.cc:738] T b8cb51676b7346df96abe11be6d053cb P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.975463 21810 consensus_queue.cc:260] T b8cb51676b7346df96abe11be6d053cb P b6ee352ce89c48d38b467042c4c3f9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.977146 21809 tablet_bootstrap.cc:492] T b8cb51676b7346df96abe11be6d053cb P 6830bca1186e4084868808884170cc33: No bootstrap required, opened a new log
I20250111 02:58:29.977586 21809 ts_tablet_manager.cc:1397] T b8cb51676b7346df96abe11be6d053cb P 6830bca1186e4084868808884170cc33: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.003s
I20250111 02:58:29.981720 21809 raft_consensus.cc:357] T b8cb51676b7346df96abe11be6d053cb P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.982460 21809 raft_consensus.cc:383] T b8cb51676b7346df96abe11be6d053cb P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:29.982698 21809 raft_consensus.cc:738] T b8cb51676b7346df96abe11be6d053cb P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830bca1186e4084868808884170cc33, State: Initialized, Role: FOLLOWER
I20250111 02:58:29.983137 21810 ts_tablet_manager.cc:1428] T b8cb51676b7346df96abe11be6d053cb P b6ee352ce89c48d38b467042c4c3f9fc: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20250111 02:58:29.983417 21809 consensus_queue.cc:260] T b8cb51676b7346df96abe11be6d053cb P 6830bca1186e4084868808884170cc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:29.997076 21809 ts_tablet_manager.cc:1428] T b8cb51676b7346df96abe11be6d053cb P 6830bca1186e4084868808884170cc33: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250111 02:58:30.001128 21821 tablet_replica.cc:331] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da: stopping tablet replica
I20250111 02:58:30.002383 21821 raft_consensus.cc:2238] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:30.003314 21821 raft_consensus.cc:2267] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:30.008628 21821 ts_tablet_manager.cc:1905] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:30.023670 21812 tablet_bootstrap.cc:492] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da: Bootstrap starting.
I20250111 02:58:30.029367 21812 tablet_bootstrap.cc:654] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:30.051766 21812 tablet_bootstrap.cc:492] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da: No bootstrap required, opened a new log
W20250111 02:58:30.051923 21793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20250111 02:58:30.052201 21812 ts_tablet_manager.cc:1397] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.000s
W20250111 02:58:30.052776 21793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:58:30.053546 21793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250111 02:58:30.055545 21812 raft_consensus.cc:357] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:30.056320 21812 raft_consensus.cc:383] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:30.056562 21812 raft_consensus.cc:738] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Initialized, Role: FOLLOWER
I20250111 02:58:30.057215 21812 consensus_queue.cc:260] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:30.078207 21812 ts_tablet_manager.cc:1428] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da: Time spent starting tablet: real 0.026s user 0.003s sys 0.003s
I20250111 02:58:30.080435 21821 ts_tablet_manager.cc:1918] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:58:30.080878 21821 log.cc:1198] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/2ba3ff8bb918447e83aba63d1357a26c
I20250111 02:58:30.081696 21821 ts_tablet_manager.cc:1939] T 2ba3ff8bb918447e83aba63d1357a26c P 2754981a32a44bf889c5d2a226e013da: Deleting consensus metadata
W20250111 02:58:30.111624 21793 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250111 02:58:30.112015 21793 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250111 02:58:30.112377 21793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 02:58:30.112777 21793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:58:30.113065 21793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 02:58:30.113307 21793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 02:58:30.181249 21793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.20.84:35279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41043
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.125:35029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.20.125
--webserver_port=38137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4120
TSAN enabled
I20250111 02:58:30.183228 21793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:58:30.185606 21793 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 02:58:30.192725 21829 raft_consensus.cc:491] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:30.193233 21829 raft_consensus.cc:513] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:30.195379 21829 leader_election.cc:290] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:30.197103 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8cb51676b7346df96abe11be6d053cb" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33" is_pre_election: true
I20250111 02:58:30.197775 21701 raft_consensus.cc:2463] T b8cb51676b7346df96abe11be6d053cb P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2754981a32a44bf889c5d2a226e013da in term 0.
I20250111 02:58:30.199404 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8cb51676b7346df96abe11be6d053cb" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" is_pre_election: true
I20250111 02:58:30.200119 21556 raft_consensus.cc:2463] T b8cb51676b7346df96abe11be6d053cb P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2754981a32a44bf889c5d2a226e013da in term 0.
I20250111 02:58:30.201711 21335 leader_election.cc:304] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2754981a32a44bf889c5d2a226e013da, b6ee352ce89c48d38b467042c4c3f9fc; no voters:
I20250111 02:58:30.203752 21829 raft_consensus.cc:2798] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:58:30.204234 21829 raft_consensus.cc:491] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:58:30.204638 21829 raft_consensus.cc:3054] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:30.207406 21844 raft_consensus.cc:491] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:30.207878 21844 raft_consensus.cc:513] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:30.209781 21844 leader_election.cc:290] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:30.212663 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c63181792b840bca29360a71def7f1c" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33" is_pre_election: true
I20250111 02:58:30.212845 21400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c63181792b840bca29360a71def7f1c" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2754981a32a44bf889c5d2a226e013da" is_pre_election: true
I20250111 02:58:30.213331 21701 raft_consensus.cc:2463] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6ee352ce89c48d38b467042c4c3f9fc in term 0.
I20250111 02:58:30.213925 21400 raft_consensus.cc:2463] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6ee352ce89c48d38b467042c4c3f9fc in term 0.
I20250111 02:58:30.214934 21492 leader_election.cc:304] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6830bca1186e4084868808884170cc33, b6ee352ce89c48d38b467042c4c3f9fc; no voters:
I20250111 02:58:30.219458 21844 raft_consensus.cc:2798] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:58:30.219902 21844 raft_consensus.cc:491] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:58:30.220294 21844 raft_consensus.cc:3054] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:58:30.221961 21882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:30.226712 21884 raft_consensus.cc:491] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:30.227402 21884 raft_consensus.cc:513] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:30.227633 21829 raft_consensus.cc:513] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:30.229614 21884 leader_election.cc:290] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:30.229713 21829 leader_election.cc:290] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 election: Requested vote from peers b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:30.229086 21844 raft_consensus.cc:513] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:30.232028 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ddafc3983b543978beb74d38cf9936e" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33" is_pre_election: true
I20250111 02:58:30.232658 21700 raft_consensus.cc:2463] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6ee352ce89c48d38b467042c4c3f9fc in term 0.
I20250111 02:58:30.233430 21844 leader_election.cc:290] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 election: Requested vote from peers 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:30.233963 21400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ddafc3983b543978beb74d38cf9936e" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2754981a32a44bf889c5d2a226e013da" is_pre_election: true
I20250111 02:58:30.234148 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8cb51676b7346df96abe11be6d053cb" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ee352ce89c48d38b467042c4c3f9fc"
I20250111 02:58:30.234634 21400 raft_consensus.cc:2463] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6ee352ce89c48d38b467042c4c3f9fc in term 0.
I20250111 02:58:30.234746 21556 raft_consensus.cc:3054] T b8cb51676b7346df96abe11be6d053cb P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:30.235531 21492 leader_election.cc:304] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6830bca1186e4084868808884170cc33, b6ee352ce89c48d38b467042c4c3f9fc; no voters:
I20250111 02:58:30.236649 21701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8cb51676b7346df96abe11be6d053cb" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33"
I20250111 02:58:30.237277 21701 raft_consensus.cc:3054] T b8cb51676b7346df96abe11be6d053cb P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:30.238057 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c63181792b840bca29360a71def7f1c" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33"
I20250111 02:58:30.238662 21700 raft_consensus.cc:3054] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:30.240252 21400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c63181792b840bca29360a71def7f1c" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2754981a32a44bf889c5d2a226e013da"
I20250111 02:58:30.240836 21400 raft_consensus.cc:3054] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:30.244671 21701 raft_consensus.cc:2463] T b8cb51676b7346df96abe11be6d053cb P 6830bca1186e4084868808884170cc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2754981a32a44bf889c5d2a226e013da in term 1.
I20250111 02:58:30.246280 21337 leader_election.cc:304] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2754981a32a44bf889c5d2a226e013da, 6830bca1186e4084868808884170cc33; no voters:
I20250111 02:58:30.247553 21829 raft_consensus.cc:2798] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:58:30.247987 21400 raft_consensus.cc:2463] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6ee352ce89c48d38b467042c4c3f9fc in term 1.
I20250111 02:58:30.249495 21489 leader_election.cc:304] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2754981a32a44bf889c5d2a226e013da, b6ee352ce89c48d38b467042c4c3f9fc; no voters:
I20250111 02:58:30.250414 21884 raft_consensus.cc:2798] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:58:30.250774 21884 raft_consensus.cc:491] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:58:30.251122 21884 raft_consensus.cc:3054] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:30.253413 21556 raft_consensus.cc:2463] T b8cb51676b7346df96abe11be6d053cb P b6ee352ce89c48d38b467042c4c3f9fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2754981a32a44bf889c5d2a226e013da in term 1.
I20250111 02:58:30.257709 21700 raft_consensus.cc:2463] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6ee352ce89c48d38b467042c4c3f9fc in term 1.
I20250111 02:58:30.259318 21884 raft_consensus.cc:513] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:30.260576 21829 raft_consensus.cc:695] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [term 1 LEADER]: Becoming Leader. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Running, Role: LEADER
I20250111 02:58:30.261598 21829 consensus_queue.cc:237] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:30.275321 21844 raft_consensus.cc:2798] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:58:30.278824 21400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ddafc3983b543978beb74d38cf9936e" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2754981a32a44bf889c5d2a226e013da"
I20250111 02:58:30.279646 21400 raft_consensus.cc:3054] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:30.281421 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ddafc3983b543978beb74d38cf9936e" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33"
I20250111 02:58:30.281950 21700 raft_consensus.cc:3054] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:30.289296 21400 raft_consensus.cc:2463] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6ee352ce89c48d38b467042c4c3f9fc in term 1.
I20250111 02:58:30.289317 21700 raft_consensus.cc:2463] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6ee352ce89c48d38b467042c4c3f9fc in term 1.
I20250111 02:58:30.290858 21489 leader_election.cc:304] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2754981a32a44bf889c5d2a226e013da, b6ee352ce89c48d38b467042c4c3f9fc; no voters:
I20250111 02:58:30.291402 21884 leader_election.cc:290] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 election: Requested vote from peers 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:30.293457 21884 raft_consensus.cc:2798] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:58:30.294566 21884 raft_consensus.cc:695] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [term 1 LEADER]: Becoming Leader. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Running, Role: LEADER
I20250111 02:58:30.296137 21884 consensus_queue.cc:237] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:30.299180 21844 raft_consensus.cc:695] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [term 1 LEADER]: Becoming Leader. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Running, Role: LEADER
I20250111 02:58:30.301723 21844 consensus_queue.cc:237] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
W20250111 02:58:30.395874 21636 connection.cc:537] server connection from 127.0.0.1:44898 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:58:30.397081 21335 connection.cc:537] server connection from 127.0.0.1:36052 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:58:30.403378 21759 connection.cc:537] client connection to 127.18.20.126:34627 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:58:30.414932 21759 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.18.20.126:34627)
W20250111 02:58:30.416337 21759 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.20.126:34627)
W20250111 02:58:30.418653 21759 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.18.20.126:34627)
W20250111 02:58:30.421550 21759 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.18.20.126:34627)
W20250111 02:58:30.423660 21637 connection.cc:537] client connection to 127.18.20.126:34627 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:58:30.424983 21337 connection.cc:537] client connection to 127.18.20.126:34627 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:58:30.425275 21492 connection.cc:537] client connection to 127.18.20.126:34627 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:58:30.439028 21489 connection.cc:537] server connection from 127.0.0.1:52636 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:58:30.441521 21808 negotiation.cc:337] Failed RPC negotiation. Trace:
0111 02:58:30.426960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.20.126:34627 (local address 127.18.20.66:39941)
0111 02:58:30.427945 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0111 02:58:30.427987 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0111 02:58:30.428260 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0111 02:58:30.440347 (+ 12087us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.20.126:34627: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":347}
W20250111 02:58:30.446156 21825 negotiation.cc:337] Failed RPC negotiation. Trace:
0111 02:58:30.426960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.20.126:34627 (local address 127.18.20.65:59333)
0111 02:58:30.427945 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0111 02:58:30.427993 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0111 02:58:30.428325 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0111 02:58:30.440382 (+ 12057us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.20.126:34627: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":347}
W20250111 02:58:30.456760 21835 negotiation.cc:337] Failed RPC negotiation. Trace:
0111 02:58:30.429052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.20.126:34627 (local address 127.18.20.67:48509)
0111 02:58:30.430234 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0111 02:58:30.430286 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0111 02:58:30.430594 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0111 02:58:30.455511 (+ 24917us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.20.126:34627: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":428}
I20250111 02:58:30.456852 21896 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3476b1c08aec44598f1c203c2cbcf016)
I20250111 02:58:30.457324 21896 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
W20250111 02:58:30.460227 21173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from unknown peer (error 108)
I20250111 02:58:30.463527 21896 leader_election.cc:290] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029)
W20250111 02:58:30.476042 21170 leader_election.cc:336] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:58:30.479756 21173 leader_election.cc:336] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627): Network error: Client connection negotiation failed: client connection to 127.18.20.126:34627: connect: Connection refused (error 111)
I20250111 02:58:30.480379 21173 leader_election.cc:304] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ebb149f387f469fbb3731becb98426f; no voters: 3476b1c08aec44598f1c203c2cbcf016, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:30.481719 21896 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250111 02:58:31.020617 21900 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3476b1c08aec44598f1c203c2cbcf016)
I20250111 02:58:31.021633 21900 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:31.024462 21900 leader_election.cc:290] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029)
W20250111 02:58:31.068042 21173 leader_election.cc:336] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627): Network error: Client connection negotiation failed: client connection to 127.18.20.126:34627: connect: Connection refused (error 111)
W20250111 02:58:31.070638 21170 leader_election.cc:336] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 02:58:31.071079 21170 leader_election.cc:304] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ebb149f387f469fbb3731becb98426f; no voters: 3476b1c08aec44598f1c203c2cbcf016, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:31.072894 21900 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250111 02:58:31.125841 21904 consensus_queue.cc:1035] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:58:31.180629 21904 consensus_queue.cc:1035] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:58:31.204198 21905 consensus_queue.cc:1035] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [LEADER]: Connected to new peer: Peer: permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:58:31.223932 21905 consensus_queue.cc:1035] T b8cb51676b7346df96abe11be6d053cb P 2754981a32a44bf889c5d2a226e013da [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:58:31.250535 21915 consensus_queue.cc:1035] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20250111 02:58:31.247879 21916 consensus_queue.cc:1035] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:58:31.825619 21930 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3476b1c08aec44598f1c203c2cbcf016)
I20250111 02:58:31.826195 21930 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:31.828912 21930 leader_election.cc:290] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029)
W20250111 02:58:31.886924 21170 leader_election.cc:336] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:58:31.900509 21173 leader_election.cc:336] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627): Network error: Client connection negotiation failed: client connection to 127.18.20.126:34627: connect: Connection refused (error 111)
I20250111 02:58:31.901089 21173 leader_election.cc:304] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ebb149f387f469fbb3731becb98426f; no voters: 3476b1c08aec44598f1c203c2cbcf016, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:31.902392 21930 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250111 02:58:31.606634 21881 debug-util.cc:398] Leaking SignalData structure 0x7b080001ca00 after lost signal to thread 21793
W20250111 02:58:32.218509 21881 kernel_stack_watchdog.cc:198] Thread 21793 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:636 for 366ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250111 02:58:30.244982 21883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:58:32.219177 21793 thread.cc:635] OpenStack (cloud detector) Time spent creating pthread: real 1.981s user 0.563s sys 1.020s
W20250111 02:58:32.219611 21793 thread.cc:601] OpenStack (cloud detector) Time spent starting thread: real 1.982s user 0.563s sys 1.020s
W20250111 02:58:32.222862 21886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:58:32.224328 21885 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1982 milliseconds
I20250111 02:58:32.224447 21793 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250111 02:58:32.225731 21793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:58:32.228945 21793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:58:32.230486 21793 hybrid_clock.cc:648] HybridClock initialized: now 1736564312230448 us; error 50 us; skew 500 ppm
I20250111 02:58:32.231613 21793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:58:32.242825 21793 webserver.cc:458] Webserver started at http://127.18.20.125:38137/ using document root <none> and password file <none>
I20250111 02:58:32.243917 21793 fs_manager.cc:362] Metadata directory not provided
I20250111 02:58:32.244212 21793 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:58:32.253651 21793 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.003s
I20250111 02:58:32.258942 21941 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:58:32.260306 21793 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250111 02:58:32.260658 21793 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/data,/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/wal
uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21"
format_stamp: "Formatted at 2025-01-11 02:58:16 on dist-test-slave-09x0"
I20250111 02:58:32.263202 21793 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:58:32.319762 21793 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:58:32.321458 21793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:58:32.321950 21793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:58:32.409547 21793 rpc_server.cc:307] RPC server started. Bound to: 127.18.20.125:35029
I20250111 02:58:32.409693 21995 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.20.125:35029 every 8 connection(s)
I20250111 02:58:32.412680 21793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-1/data/info.pb
I20250111 02:58:32.415948 18513 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu as pid 21793
I20250111 02:58:32.430402 21996 sys_catalog.cc:263] Verifying existing consensus state
I20250111 02:58:32.441527 21996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: Bootstrap starting.
I20250111 02:58:32.458266 21996 log.cc:826] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: Log is configured to *not* fsync() on all Append() calls
I20250111 02:58:32.475658 21996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:58:32.476644 21996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: Bootstrap complete.
I20250111 02:58:32.497702 21996 raft_consensus.cc:357] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:32.500429 21996 raft_consensus.cc:738] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3a224ebe0341df84e1d1d8fdfb0f21, State: Initialized, Role: FOLLOWER
I20250111 02:58:32.501327 21996 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:32.506297 22002 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:32.507298 22002 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:32.507683 21996 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:58:32.542950 22017 catalog_manager.cc:1260] Loaded cluster ID: 3cbb96e3d31f437a849351b045304259
I20250111 02:58:32.543277 22017 catalog_manager.cc:1553] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: loading cluster ID for follower catalog manager: success
I20250111 02:58:32.552731 22017 catalog_manager.cc:1575] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: acquiring CA information for follower catalog manager: success
I20250111 02:58:32.559243 22017 catalog_manager.cc:1603] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 02:58:32.545760 21793 master_runner.cc:186] Error getting master registration for 127.18.20.126:34627: Network error: Client connection negotiation failed: client connection to 127.18.20.126:34627: connect: Connection refused (error 111),
I20250111 02:58:32.575767 21793 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:58:32.751001 22020 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3476b1c08aec44598f1c203c2cbcf016)
I20250111 02:58:32.751526 22020 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:32.753562 22020 leader_election.cc:290] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029)
W20250111 02:58:32.766693 21173 leader_election.cc:336] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627): Network error: Client connection negotiation failed: client connection to 127.18.20.126:34627: connect: Connection refused (error 111)
I20250111 02:58:32.769940 21971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 6 candidate_status { last_received { term: 5 index: 26 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" is_pre_election: true
I20250111 02:58:32.771114 21971 raft_consensus.cc:2463] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ebb149f387f469fbb3731becb98426f in term 4.
I20250111 02:58:32.772452 21170 leader_election.cc:304] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21; no voters: 3476b1c08aec44598f1c203c2cbcf016
I20250111 02:58:32.773344 22020 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250111 02:58:32.773679 22020 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3476b1c08aec44598f1c203c2cbcf016)
I20250111 02:58:32.773955 22020 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 5 FOLLOWER]: Advancing to term 6
I20250111 02:58:32.780196 22020 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:32.781878 22020 leader_election.cc:290] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 election: Requested vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029)
I20250111 02:58:32.783313 21971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ebb149f387f469fbb3731becb98426f" candidate_term: 6 candidate_status { last_received { term: 5 index: 26 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21"
I20250111 02:58:32.783937 21971 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 4 FOLLOWER]: Advancing to term 6
W20250111 02:58:32.786094 21173 leader_election.cc:336] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627): Network error: Client connection negotiation failed: client connection to 127.18.20.126:34627: connect: Connection refused (error 111)
I20250111 02:58:32.795660 21971 raft_consensus.cc:2463] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ebb149f387f469fbb3731becb98426f in term 6.
I20250111 02:58:32.813807 21170 leader_election.cc:304] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21; no voters: 3476b1c08aec44598f1c203c2cbcf016
I20250111 02:58:32.815132 22020 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 6 FOLLOWER]: Leader election won for term 6
I20250111 02:58:32.817545 22020 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [term 6 LEADER]: Becoming Leader. State: Replica: 7ebb149f387f469fbb3731becb98426f, State: Running, Role: LEADER
I20250111 02:58:32.818768 22020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 5.26, Last appended by leader: 26, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:32.837711 22026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ebb149f387f469fbb3731becb98426f. Latest consensus state: current_term: 6 leader_uuid: "7ebb149f387f469fbb3731becb98426f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:32.838650 22026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: This master's current role is: LEADER
I20250111 02:58:32.843302 21602 heartbeater.cc:346] Connected to a master server at 127.18.20.125:35029
I20250111 02:58:32.854028 21961 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" instance_seqno: 1736564304600135) as {username='slave'} at 127.18.20.66:54431; Asking this server to re-register.
I20250111 02:58:32.861482 22028 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:58:32.873447 21747 heartbeater.cc:346] Connected to a master server at 127.18.20.125:35029
I20250111 02:58:32.874024 21747 heartbeater.cc:502] Master 127.18.20.125:35029 was elected leader, sending a full tablet report...
I20250111 02:58:32.874758 21457 heartbeater.cc:346] Connected to a master server at 127.18.20.125:35029
I20250111 02:58:32.878130 22028 catalog_manager.cc:670] Loaded metadata for table default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]
I20250111 02:58:32.880192 22028 catalog_manager.cc:670] Loaded metadata for table default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]
I20250111 02:58:32.882081 22028 catalog_manager.cc:670] Loaded metadata for table default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]
I20250111 02:58:32.885502 21960 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6830bca1186e4084868808884170cc33" instance_seqno: 1736564306843221) as {username='slave'} at 127.18.20.67:56883; Asking this server to re-register.
I20250111 02:58:32.888288 21959 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2754981a32a44bf889c5d2a226e013da" instance_seqno: 1736564302339577) as {username='slave'} at 127.18.20.65:52155; Asking this server to re-register.
I20250111 02:58:32.890756 21747 heartbeater.cc:463] Registering TS with master...
I20250111 02:58:32.891702 21457 heartbeater.cc:463] Registering TS with master...
I20250111 02:58:32.896299 21960 ts_manager.cc:194] Registered new tserver with Master: 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:32.898025 21959 ts_manager.cc:194] Registered new tserver with Master: 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021)
I20250111 02:58:32.909873 22028 tablet_loader.cc:96] loaded metadata for tablet 1ddafc3983b543978beb74d38cf9936e (table default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f])
I20250111 02:58:32.912861 21602 heartbeater.cc:463] Registering TS with master...
I20250111 02:58:32.913347 22028 tablet_loader.cc:96] loaded metadata for tablet 9c63181792b840bca29360a71def7f1c (table default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4])
I20250111 02:58:32.914793 22028 tablet_loader.cc:96] loaded metadata for tablet b8cb51676b7346df96abe11be6d053cb (table default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb])
I20250111 02:58:32.917163 21960 ts_manager.cc:194] Registered new tserver with Master: b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313)
I20250111 02:58:32.926064 22028 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:58:32.930761 22028 catalog_manager.cc:1260] Loaded cluster ID: 3cbb96e3d31f437a849351b045304259
I20250111 02:58:32.931020 22028 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:58:32.935103 22028 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:58:32.939491 22028 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: Loaded TSK: 0
I20250111 02:58:32.940372 22028 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:58:32.943881 22028 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 02:58:32.947788 22028 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250111 02:58:32.961311 21971 raft_consensus.cc:1270] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Refusing update from remote peer 7ebb149f387f469fbb3731becb98426f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 28. (index mismatch)
I20250111 02:58:32.963212 22020 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 5, Time since last communication: 0.000s
W20250111 02:58:32.966993 21173 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f -> Peer 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627): Couldn't send request to peer 3476b1c08aec44598f1c203c2cbcf016. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.126:34627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:58:32.996949 21186 master_service.cc:946] ReplaceTablet: received request to replace tablet 1ddafc3983b543978beb74d38cf9936e from {username='slave'} at 127.0.0.1:56564
I20250111 02:58:32.996518 21184 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56564:
name: "default.Table_dd8c5f570a5c46a4a461cb9d3c409afe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:32.997093 21183 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56564:
name: "default.Table_0939bec9a16f450fa14248fcc1cd2fc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:32.999939 21182 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56564:
table { table_name: "default.Table_46ab12e3d44441d982fda4c1aba20a2d" } new_table_name: "default.Table_77a9864685c248e9bb93b3b1f75adb6e" modify_external_catalogs: true
I20250111 02:58:33.001832 21179 master_service.cc:946] ReplaceTablet: received request to replace tablet 9c63181792b840bca29360a71def7f1c from {username='slave'} at 127.0.0.1:56564
I20250111 02:58:33.003161 21178 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56564:
name: "default.Table_95e9bf8a21a4474781705bea1e0c7b91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:33.005530 21180 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56564:
name: "default.Table_b371b664c5b043be8b2f0353a9e57e8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:33.010149 21181 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56564:
table { table_name: "default.Table_15ebf4fa41db452c957f7aa50ddfc192" } new_table_name: "default.Table_7e8389e7a1b745a8898f6a0862a97327" modify_external_catalogs: true
F20250111 02:58:33.011124 21243 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_95e9bf8a21a4474781705bea1e0c7b91
*** Check failure stack trace: ***
I20250111 02:58:32.999264 21185 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56564:
table { table_name: "default.Table_836e7aa15e09458c9659cb4a8b7d352a" } modify_external_catalogs: true
I20250111 02:58:33.015045 21185 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56564:
table { table_name: "default.Table_836e7aa15e09458c9659cb4a8b7d352a" } modify_external_catalogs: true
@ 0x7f537ce48950 google::LogMessage::Flush() at ??:0
@ 0x7f537ce486bf google::LogMessage::~LogMessage() at ??:0
I20250111 02:58:33.077092 22002 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ebb149f387f469fbb3731becb98426f. Latest consensus state: current_term: 6 leader_uuid: "7ebb149f387f469fbb3731becb98426f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:33.077935 22002 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:33.082873 22035 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "7ebb149f387f469fbb3731becb98426f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:33.084084 22035 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f5387d4382a kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250111 02:58:33.101107 22035 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "7ebb149f387f469fbb3731becb98426f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:33.101898 22035 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:33.105027 22020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "7ebb149f387f469fbb3731becb98426f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:33.105957 22020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f [sys.catalog]: This master's current role is: LEADER
I20250111 02:58:33.110163 21179 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: Sending DeleteTablet for 3 replicas of tablet 9c63181792b840bca29360a71def7f1c
I20250111 02:58:33.114634 21186 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: Sending DeleteTablet for 3 replicas of tablet 1ddafc3983b543978beb74d38cf9936e
I20250111 02:58:33.117056 21186 catalog_manager.cc:6596] ReplaceTablet: tablet 1ddafc3983b543978beb74d38cf9936e deleted and replaced by tablet dacdeedd62a944819edbf93c9f57d269
I20250111 02:58:33.121970 21179 catalog_manager.cc:6596] ReplaceTablet: tablet 9c63181792b840bca29360a71def7f1c deleted and replaced by tablet 0d09de52e0dd44f79b706c6aef6cdb42
@ 0x7f5387d4307c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250111 02:58:33.174640 22032 mvcc.cc:204] Tried to move back new op lower bound from 7112967425874018304 to 7112967403366813696. Current Snapshot: MvccSnapshot[applied={T|T < 7112967425874018304 or (T in {7112967425874018304})}]
I20250111 02:58:33.206511 21379 tablet_service.cc:1467] Processing CreateTablet for tablet 0d09de52e0dd44f79b706c6aef6cdb42 (DEFAULT_TABLE table=default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:33.207800 21379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d09de52e0dd44f79b706c6aef6cdb42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:33.208906 21380 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c63181792b840bca29360a71def7f1c with delete_type TABLET_DATA_DELETED (replaced by tablet 0d09de52e0dd44f79b706c6aef6cdb42) from {username='slave'} at 127.0.0.1:36064
I20250111 02:58:33.209822 21381 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ddafc3983b543978beb74d38cf9936e with delete_type TABLET_DATA_DELETED (replaced by tablet dacdeedd62a944819edbf93c9f57d269) from {username='slave'} at 127.0.0.1:36064
@ 0x7f5387d46d32 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250111 02:58:33.211364 21378 tablet_service.cc:1467] Processing CreateTablet for tablet dacdeedd62a944819edbf93c9f57d269 (DEFAULT_TABLE table=default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:33.212610 21378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dacdeedd62a944819edbf93c9f57d269. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:33.214480 21678 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ddafc3983b543978beb74d38cf9936e with delete_type TABLET_DATA_DELETED (replaced by tablet dacdeedd62a944819edbf93c9f57d269) from {username='slave'} at 127.0.0.1:44906
I20250111 02:58:33.215760 21680 tablet_service.cc:1467] Processing CreateTablet for tablet 0d09de52e0dd44f79b706c6aef6cdb42 (DEFAULT_TABLE table=default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:33.216518 21679 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c63181792b840bca29360a71def7f1c with delete_type TABLET_DATA_DELETED (replaced by tablet 0d09de52e0dd44f79b706c6aef6cdb42) from {username='slave'} at 127.0.0.1:44906
I20250111 02:58:33.216926 21680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d09de52e0dd44f79b706c6aef6cdb42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:33.217742 21681 tablet_service.cc:1467] Processing CreateTablet for tablet dacdeedd62a944819edbf93c9f57d269 (DEFAULT_TABLE table=default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:33.218812 21681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dacdeedd62a944819edbf93c9f57d269. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:33.223953 22048 tablet_replica.cc:331] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33: stopping tablet replica
I20250111 02:58:33.224830 22048 raft_consensus.cc:2238] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:33.225368 22048 raft_consensus.cc:2267] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:33.227573 22048 ts_tablet_manager.cc:1905] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:33.243160 22048 ts_tablet_manager.cc:1918] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:58:33.243559 22048 log.cc:1198] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/1ddafc3983b543978beb74d38cf9936e
I20250111 02:58:33.244374 22048 ts_tablet_manager.cc:1939] T 1ddafc3983b543978beb74d38cf9936e P 6830bca1186e4084868808884170cc33: Deleting consensus metadata
I20250111 02:58:33.247352 21173 catalog_manager.cc:4872] TS 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): tablet 1ddafc3983b543978beb74d38cf9936e (table default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]) successfully deleted
I20250111 02:58:33.249063 22048 tablet_replica.cc:331] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33: stopping tablet replica
I20250111 02:58:33.250061 22048 raft_consensus.cc:2238] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:33.250666 22048 raft_consensus.cc:2267] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:33.252836 22048 ts_tablet_manager.cc:1905] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:33.270337 22048 ts_tablet_manager.cc:1918] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:58:33.270730 22048 log.cc:1198] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-2/wal/wals/9c63181792b840bca29360a71def7f1c
I20250111 02:58:33.271638 22048 ts_tablet_manager.cc:1939] T 9c63181792b840bca29360a71def7f1c P 6830bca1186e4084868808884170cc33: Deleting consensus metadata
I20250111 02:58:33.274557 21173 catalog_manager.cc:4872] TS 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): tablet 9c63181792b840bca29360a71def7f1c (table default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]) successfully deleted
I20250111 02:58:33.277271 22050 tablet_bootstrap.cc:492] T 0d09de52e0dd44f79b706c6aef6cdb42 P 6830bca1186e4084868808884170cc33: Bootstrap starting.
I20250111 02:58:33.279098 21535 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ddafc3983b543978beb74d38cf9936e with delete_type TABLET_DATA_DELETED (replaced by tablet dacdeedd62a944819edbf93c9f57d269) from {username='slave'} at 127.0.0.1:52642
I20250111 02:58:33.280396 21536 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c63181792b840bca29360a71def7f1c with delete_type TABLET_DATA_DELETED (replaced by tablet 0d09de52e0dd44f79b706c6aef6cdb42) from {username='slave'} at 127.0.0.1:52642
I20250111 02:58:33.281937 21534 tablet_service.cc:1467] Processing CreateTablet for tablet 0d09de52e0dd44f79b706c6aef6cdb42 (DEFAULT_TABLE table=default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:33.284091 21533 tablet_service.cc:1467] Processing CreateTablet for tablet dacdeedd62a944819edbf93c9f57d269 (DEFAULT_TABLE table=default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:33.285349 21533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dacdeedd62a944819edbf93c9f57d269. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:33.287393 21534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d09de52e0dd44f79b706c6aef6cdb42. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f5387d46cea _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20250111 02:58:33.294054 22047 tablet_replica.cc:331] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da: stopping tablet replica
I20250111 02:58:33.295162 22047 raft_consensus.cc:2238] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:33.295684 22047 raft_consensus.cc:2267] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:33.302899 22050 tablet_bootstrap.cc:654] T 0d09de52e0dd44f79b706c6aef6cdb42 P 6830bca1186e4084868808884170cc33: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:33.310318 22047 ts_tablet_manager.cc:1905] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:33.324157 22051 tablet_replica.cc:331] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc: stopping tablet replica
I20250111 02:58:33.325137 22051 raft_consensus.cc:2238] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:58:33.326768 22051 raft_consensus.cc:2267] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:33.327216 21179 master_service.cc:946] ReplaceTablet: received request to replace tablet 1ddafc3983b543978beb74d38cf9936e from {username='slave'} at 127.0.0.1:56564
I20250111 02:58:33.328513 22052 tablet_bootstrap.cc:492] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da: Bootstrap starting.
I20250111 02:58:33.331806 21179 master_service.cc:946] ReplaceTablet: received request to replace tablet 0d09de52e0dd44f79b706c6aef6cdb42 from {username='slave'} at 127.0.0.1:56564
I20250111 02:58:33.341953 22051 ts_tablet_manager.cc:1905] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:33.352614 21179 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 7ebb149f387f469fbb3731becb98426f: Sending DeleteTablet for 3 replicas of tablet 0d09de52e0dd44f79b706c6aef6cdb42
I20250111 02:58:33.354694 21535 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d09de52e0dd44f79b706c6aef6cdb42 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ac64b6a72274b36a22727ae79227223) from {username='slave'} at 127.0.0.1:52642
I20250111 02:58:33.355352 21681 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d09de52e0dd44f79b706c6aef6cdb42 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ac64b6a72274b36a22727ae79227223) from {username='slave'} at 127.0.0.1:44906
I20250111 02:58:33.355710 21379 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d09de52e0dd44f79b706c6aef6cdb42 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ac64b6a72274b36a22727ae79227223) from {username='slave'} at 127.0.0.1:36064
I20250111 02:58:33.358983 22056 tablet_bootstrap.cc:492] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc: Bootstrap starting.
W20250111 02:58:33.360085 21173 catalog_manager.cc:4852] TS 6830bca1186e4084868808884170cc33 (127.18.20.67:43789): delete failed for tablet 0d09de52e0dd44f79b706c6aef6cdb42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d09de52e0dd44f79b706c6aef6cdb42 already in progress: creating tablet
I20250111 02:58:33.362201 21179 catalog_manager.cc:6596] ReplaceTablet: tablet 0d09de52e0dd44f79b706c6aef6cdb42 deleted and replaced by tablet 9ac64b6a72274b36a22727ae79227223
I20250111 02:58:33.366119 22056 tablet_bootstrap.cc:654] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:33.369800 22050 tablet_bootstrap.cc:492] T 0d09de52e0dd44f79b706c6aef6cdb42 P 6830bca1186e4084868808884170cc33: No bootstrap required, opened a new log
I20250111 02:58:33.370322 22050 ts_tablet_manager.cc:1397] T 0d09de52e0dd44f79b706c6aef6cdb42 P 6830bca1186e4084868808884170cc33: Time spent bootstrapping tablet: real 0.093s user 0.026s sys 0.005s
I20250111 02:58:33.373952 22050 raft_consensus.cc:357] T 0d09de52e0dd44f79b706c6aef6cdb42 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } }
I20250111 02:58:33.374779 22050 raft_consensus.cc:383] T 0d09de52e0dd44f79b706c6aef6cdb42 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:33.375152 22050 raft_consensus.cc:738] T 0d09de52e0dd44f79b706c6aef6cdb42 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830bca1186e4084868808884170cc33, State: Initialized, Role: FOLLOWER
I20250111 02:58:33.375902 22050 consensus_queue.cc:260] T 0d09de52e0dd44f79b706c6aef6cdb42 P 6830bca1186e4084868808884170cc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } }
I20250111 02:58:33.381152 22051 ts_tablet_manager.cc:1918] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:58:33.381682 22051 log.cc:1198] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/1ddafc3983b543978beb74d38cf9936e
I20250111 02:58:33.382695 22051 ts_tablet_manager.cc:1939] T 1ddafc3983b543978beb74d38cf9936e P b6ee352ce89c48d38b467042c4c3f9fc: Deleting consensus metadata
I20250111 02:58:33.386047 21171 catalog_manager.cc:4872] TS b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313): tablet 1ddafc3983b543978beb74d38cf9936e (table default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]) successfully deleted
I20250111 02:58:33.387194 22051 tablet_replica.cc:331] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc: stopping tablet replica
I20250111 02:58:33.388139 22051 raft_consensus.cc:2238] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:58:33.389204 22051 raft_consensus.cc:2267] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:33.390321 22056 tablet_bootstrap.cc:492] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc: No bootstrap required, opened a new log
I20250111 02:58:33.390795 22056 ts_tablet_manager.cc:1397] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.009s
I20250111 02:58:33.391650 22051 ts_tablet_manager.cc:1905] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:33.398722 22056 raft_consensus.cc:357] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.399891 22056 raft_consensus.cc:383] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:33.408169 22052 tablet_bootstrap.cc:654] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:33.415963 21746 heartbeater.cc:502] Master 127.18.20.124:36707 was elected leader, sending a full tablet report...
I20250111 02:58:33.424198 22047 ts_tablet_manager.cc:1918] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:58:33.424651 22047 log.cc:1198] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/9c63181792b840bca29360a71def7f1c
I20250111 02:58:33.425490 22047 ts_tablet_manager.cc:1939] T 9c63181792b840bca29360a71def7f1c P 2754981a32a44bf889c5d2a226e013da: Deleting consensus metadata
@ 0x7f5387d46c7a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20250111 02:58:33.437888 22047 tablet_replica.cc:331] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da: stopping tablet replica
I20250111 02:58:33.438642 22047 raft_consensus.cc:2238] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:58:33.438575 21170 catalog_manager.cc:4872] TS 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): tablet 9c63181792b840bca29360a71def7f1c (table default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]) successfully deleted
I20250111 02:58:33.439152 22047 raft_consensus.cc:2267] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:58:33.439680 21379 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac64b6a72274b36a22727ae79227223 (DEFAULT_TABLE table=default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:33.440837 21379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac64b6a72274b36a22727ae79227223. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:33.441102 22047 ts_tablet_manager.cc:1905] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:58:33.447093 22050 ts_tablet_manager.cc:1428] T 0d09de52e0dd44f79b706c6aef6cdb42 P 6830bca1186e4084868808884170cc33: Time spent starting tablet: real 0.076s user 0.002s sys 0.017s
I20250111 02:58:33.446923 21681 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac64b6a72274b36a22727ae79227223 (DEFAULT_TABLE table=default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:33.448145 21681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac64b6a72274b36a22727ae79227223. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:33.450018 22050 tablet_bootstrap.cc:492] T dacdeedd62a944819edbf93c9f57d269 P 6830bca1186e4084868808884170cc33: Bootstrap starting.
I20250111 02:58:33.407434 22051 ts_tablet_manager.cc:1918] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:58:33.445777 21533 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac64b6a72274b36a22727ae79227223 (DEFAULT_TABLE table=default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:33.456670 22056 raft_consensus.cc:738] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Initialized, Role: FOLLOWER
I20250111 02:58:33.456811 22051 log.cc:1198] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-1/wal/wals/9c63181792b840bca29360a71def7f1c
I20250111 02:58:33.457448 21533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac64b6a72274b36a22727ae79227223. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:33.457674 22051 ts_tablet_manager.cc:1939] T 9c63181792b840bca29360a71def7f1c P b6ee352ce89c48d38b467042c4c3f9fc: Deleting consensus metadata
I20250111 02:58:33.443506 21179 catalog_manager.cc:5526] T b8cb51676b7346df96abe11be6d053cb P 6830bca1186e4084868808884170cc33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2754981a32a44bf889c5d2a226e013da (127.18.20.65). New cstate: current_term: 1 leader_uuid: "2754981a32a44bf889c5d2a226e013da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } } }
I20250111 02:58:33.460666 21171 catalog_manager.cc:4872] TS b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313): tablet 9c63181792b840bca29360a71def7f1c (table default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]) successfully deleted
W20250111 02:58:33.461797 21171 catalog_manager.cc:4852] TS b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313): delete failed for tablet 0d09de52e0dd44f79b706c6aef6cdb42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d09de52e0dd44f79b706c6aef6cdb42 already in progress: creating tablet
I20250111 02:58:33.461930 22056 consensus_queue.cc:260] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.463685 21601 heartbeater.cc:502] Master 127.18.20.124:36707 was elected leader, sending a full tablet report...
I20250111 02:58:33.464514 22056 ts_tablet_manager.cc:1428] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc: Time spent starting tablet: real 0.073s user 0.007s sys 0.000s
I20250111 02:58:33.465567 22056 tablet_bootstrap.cc:492] T 0d09de52e0dd44f79b706c6aef6cdb42 P b6ee352ce89c48d38b467042c4c3f9fc: Bootstrap starting.
I20250111 02:58:33.456347 22050 tablet_bootstrap.cc:654] T dacdeedd62a944819edbf93c9f57d269 P 6830bca1186e4084868808884170cc33: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:33.476195 22047 ts_tablet_manager.cc:1918] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:58:33.480980 22056 tablet_bootstrap.cc:654] T 0d09de52e0dd44f79b706c6aef6cdb42 P b6ee352ce89c48d38b467042c4c3f9fc: Neither blocks nor log segments found. Creating new log.
@ 0x7f5387d46c42 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20250111 02:58:33.476583 22047 log.cc:1198] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da: Deleting WAL directory at /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/ts-0/wal/wals/1ddafc3983b543978beb74d38cf9936e
I20250111 02:58:33.484588 22047 ts_tablet_manager.cc:1939] T 1ddafc3983b543978beb74d38cf9936e P 2754981a32a44bf889c5d2a226e013da: Deleting consensus metadata
I20250111 02:58:33.489557 21170 catalog_manager.cc:4872] TS 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): tablet 1ddafc3983b543978beb74d38cf9936e (table default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]) successfully deleted
W20250111 02:58:33.500763 21170 catalog_manager.cc:4852] TS 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): delete failed for tablet 0d09de52e0dd44f79b706c6aef6cdb42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d09de52e0dd44f79b706c6aef6cdb42 already in progress: creating tablet
I20250111 02:58:33.502802 22050 tablet_bootstrap.cc:492] T dacdeedd62a944819edbf93c9f57d269 P 6830bca1186e4084868808884170cc33: No bootstrap required, opened a new log
I20250111 02:58:33.503309 22050 ts_tablet_manager.cc:1397] T dacdeedd62a944819edbf93c9f57d269 P 6830bca1186e4084868808884170cc33: Time spent bootstrapping tablet: real 0.053s user 0.014s sys 0.000s
I20250111 02:58:33.506055 22050 raft_consensus.cc:357] T dacdeedd62a944819edbf93c9f57d269 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.506835 22050 raft_consensus.cc:383] T dacdeedd62a944819edbf93c9f57d269 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:33.507144 22050 raft_consensus.cc:738] T dacdeedd62a944819edbf93c9f57d269 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830bca1186e4084868808884170cc33, State: Initialized, Role: FOLLOWER
I20250111 02:58:33.507964 22050 consensus_queue.cc:260] T dacdeedd62a944819edbf93c9f57d269 P 6830bca1186e4084868808884170cc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
@ 0x7f5387d45f3e std::__1::__function::__func<>::operator()() at ??:0
@ 0x59ea05 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x59e83a std::__1::function<>::operator()() at ??:?
I20250111 02:58:33.521667 21533 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d09de52e0dd44f79b706c6aef6cdb42 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ac64b6a72274b36a22727ae79227223) from {username='slave'} at 127.0.0.1:52642
W20250111 02:58:33.523805 21171 catalog_manager.cc:4852] TS b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313): delete failed for tablet 0d09de52e0dd44f79b706c6aef6cdb42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d09de52e0dd44f79b706c6aef6cdb42 already in progress: creating tablet
I20250111 02:58:33.524762 22050 ts_tablet_manager.cc:1428] T dacdeedd62a944819edbf93c9f57d269 P 6830bca1186e4084868808884170cc33: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20250111 02:58:33.535982 21186 master_service.cc:946] ReplaceTablet: received request to replace tablet b8cb51676b7346df96abe11be6d053cb from {username='slave'} at 127.0.0.1:56564
@ 0x7f537f08d8d7 kudu::Thread::SuperviseThread() at ??:0
@ 0x468a00 __tsan_thread_start_func at ??:?
@ 0x7f538070c6db start_thread at ??:0
@ 0x7f537baed71f clone at ??:0
I20250111 02:58:33.593284 22052 tablet_bootstrap.cc:492] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da: No bootstrap required, opened a new log
I20250111 02:58:33.594837 22052 ts_tablet_manager.cc:1397] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da: Time spent bootstrapping tablet: real 0.267s user 0.008s sys 0.002s
I20250111 02:58:33.599504 22056 tablet_bootstrap.cc:492] T 0d09de52e0dd44f79b706c6aef6cdb42 P b6ee352ce89c48d38b467042c4c3f9fc: No bootstrap required, opened a new log
I20250111 02:58:33.597855 22052 raft_consensus.cc:357] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } }
I20250111 02:58:33.599792 22052 raft_consensus.cc:383] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:33.599973 22056 ts_tablet_manager.cc:1397] T 0d09de52e0dd44f79b706c6aef6cdb42 P b6ee352ce89c48d38b467042c4c3f9fc: Time spent bootstrapping tablet: real 0.135s user 0.022s sys 0.020s
I20250111 02:58:33.600081 22052 raft_consensus.cc:738] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Initialized, Role: FOLLOWER
I20250111 02:58:33.601918 22052 consensus_queue.cc:260] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } }
I20250111 02:58:33.604944 21456 heartbeater.cc:502] Master 127.18.20.124:36707 was elected leader, sending a full tablet report...
I20250111 02:58:33.609558 22056 raft_consensus.cc:357] T 0d09de52e0dd44f79b706c6aef6cdb42 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } }
I20250111 02:58:33.610383 22056 raft_consensus.cc:383] T 0d09de52e0dd44f79b706c6aef6cdb42 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:33.610685 22056 raft_consensus.cc:738] T 0d09de52e0dd44f79b706c6aef6cdb42 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Initialized, Role: FOLLOWER
I20250111 02:58:33.611387 22056 consensus_queue.cc:260] T 0d09de52e0dd44f79b706c6aef6cdb42 P b6ee352ce89c48d38b467042c4c3f9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } }
I20250111 02:58:33.621758 22052 ts_tablet_manager.cc:1428] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da: Time spent starting tablet: real 0.026s user 0.005s sys 0.001s
I20250111 02:58:33.624995 22055 raft_consensus.cc:491] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:33.625542 22056 ts_tablet_manager.cc:1428] T 0d09de52e0dd44f79b706c6aef6cdb42 P b6ee352ce89c48d38b467042c4c3f9fc: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20250111 02:58:33.625489 22055 raft_consensus.cc:513] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } }
I20250111 02:58:33.627559 22055 leader_election.cc:290] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:33.628479 22050 tablet_bootstrap.cc:492] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33: Bootstrap starting.
I20250111 02:58:33.633313 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d09de52e0dd44f79b706c6aef6cdb42" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" is_pre_election: true
I20250111 02:58:33.634147 21556 raft_consensus.cc:2463] T 0d09de52e0dd44f79b706c6aef6cdb42 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2754981a32a44bf889c5d2a226e013da in term 0.
I20250111 02:58:33.635373 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d09de52e0dd44f79b706c6aef6cdb42" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33" is_pre_election: true
I20250111 02:58:33.636073 21700 raft_consensus.cc:2463] T 0d09de52e0dd44f79b706c6aef6cdb42 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2754981a32a44bf889c5d2a226e013da in term 0.
I20250111 02:58:33.637174 22056 tablet_bootstrap.cc:492] T 9ac64b6a72274b36a22727ae79227223 P b6ee352ce89c48d38b467042c4c3f9fc: Bootstrap starting.
I20250111 02:58:33.637434 21337 leader_election.cc:304] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2754981a32a44bf889c5d2a226e013da, 6830bca1186e4084868808884170cc33; no voters:
I20250111 02:58:33.642191 22050 tablet_bootstrap.cc:654] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:33.643302 22055 raft_consensus.cc:2798] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:58:33.643647 22055 raft_consensus.cc:491] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:58:33.643970 22055 raft_consensus.cc:3054] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:33.652279 22056 tablet_bootstrap.cc:654] T 9ac64b6a72274b36a22727ae79227223 P b6ee352ce89c48d38b467042c4c3f9fc: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:33.660423 21942 raft_consensus.cc:491] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:33.660946 21942 raft_consensus.cc:513] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.663252 21942 leader_election.cc:290] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:33.664235 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacdeedd62a944819edbf93c9f57d269" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2754981a32a44bf889c5d2a226e013da" is_pre_election: true
I20250111 02:58:33.664374 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacdeedd62a944819edbf93c9f57d269" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33" is_pre_election: true
I20250111 02:58:33.665037 21700 raft_consensus.cc:2463] T dacdeedd62a944819edbf93c9f57d269 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6ee352ce89c48d38b467042c4c3f9fc in term 0.
W20250111 02:58:33.665877 21489 leader_election.cc:343] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:58:33.666601 21492 leader_election.cc:304] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6830bca1186e4084868808884170cc33, b6ee352ce89c48d38b467042c4c3f9fc; no voters: 2754981a32a44bf889c5d2a226e013da
I20250111 02:58:33.659360 22052 tablet_bootstrap.cc:492] T dacdeedd62a944819edbf93c9f57d269 P 2754981a32a44bf889c5d2a226e013da: Bootstrap starting.
I20250111 02:58:33.667776 21942 raft_consensus.cc:2798] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:58:33.668113 21942 raft_consensus.cc:491] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:58:33.668413 21942 raft_consensus.cc:3054] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:33.673647 22052 tablet_bootstrap.cc:654] T dacdeedd62a944819edbf93c9f57d269 P 2754981a32a44bf889c5d2a226e013da: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:33.675328 22055 raft_consensus.cc:513] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } }
I20250111 02:58:33.675683 21942 raft_consensus.cc:513] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.678354 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d09de52e0dd44f79b706c6aef6cdb42" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ee352ce89c48d38b467042c4c3f9fc"
I20250111 02:58:33.678637 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d09de52e0dd44f79b706c6aef6cdb42" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33"
I20250111 02:58:33.678979 21556 raft_consensus.cc:3054] T 0d09de52e0dd44f79b706c6aef6cdb42 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:33.679273 21700 raft_consensus.cc:3054] T 0d09de52e0dd44f79b706c6aef6cdb42 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:33.680884 22050 tablet_bootstrap.cc:492] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33: No bootstrap required, opened a new log
I20250111 02:58:33.681345 22050 ts_tablet_manager.cc:1397] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.000s
I20250111 02:58:33.683210 22055 leader_election.cc:290] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 election: Requested vote from peers b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:33.684706 22050 raft_consensus.cc:357] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.685454 22050 raft_consensus.cc:383] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:33.685722 22050 raft_consensus.cc:738] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830bca1186e4084868808884170cc33, State: Initialized, Role: FOLLOWER
I20250111 02:58:33.685947 21556 raft_consensus.cc:2463] T 0d09de52e0dd44f79b706c6aef6cdb42 P b6ee352ce89c48d38b467042c4c3f9fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2754981a32a44bf889c5d2a226e013da in term 1.
I20250111 02:58:33.686446 22050 consensus_queue.cc:260] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.687191 21335 leader_election.cc:304] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2754981a32a44bf889c5d2a226e013da, b6ee352ce89c48d38b467042c4c3f9fc; no voters:
I20250111 02:58:33.687969 21700 raft_consensus.cc:2463] T 0d09de52e0dd44f79b706c6aef6cdb42 P 6830bca1186e4084868808884170cc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2754981a32a44bf889c5d2a226e013da in term 1.
I20250111 02:58:33.688136 22055 raft_consensus.cc:2798] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:58:33.693552 22055 raft_consensus.cc:695] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [term 1 LEADER]: Becoming Leader. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Running, Role: LEADER
I20250111 02:58:33.694424 22055 consensus_queue.cc:237] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } }
I20250111 02:58:33.697598 22056 tablet_bootstrap.cc:492] T 9ac64b6a72274b36a22727ae79227223 P b6ee352ce89c48d38b467042c4c3f9fc: No bootstrap required, opened a new log
I20250111 02:58:33.698139 22056 ts_tablet_manager.cc:1397] T 9ac64b6a72274b36a22727ae79227223 P b6ee352ce89c48d38b467042c4c3f9fc: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.003s
I20250111 02:58:33.699311 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacdeedd62a944819edbf93c9f57d269" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2754981a32a44bf889c5d2a226e013da"
W20250111 02:58:33.700721 21489 leader_election.cc:343] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:58:33.700886 22056 raft_consensus.cc:357] T 9ac64b6a72274b36a22727ae79227223 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.701660 22056 raft_consensus.cc:383] T 9ac64b6a72274b36a22727ae79227223 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:33.701933 22056 raft_consensus.cc:738] T 9ac64b6a72274b36a22727ae79227223 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Initialized, Role: FOLLOWER
I20250111 02:58:33.703315 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacdeedd62a944819edbf93c9f57d269" candidate_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33"
I20250111 02:58:33.703904 21700 raft_consensus.cc:3054] T dacdeedd62a944819edbf93c9f57d269 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:33.704123 22050 ts_tablet_manager.cc:1428] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33: Time spent starting tablet: real 0.022s user 0.007s sys 0.001s
I20250111 02:58:33.706140 22056 consensus_queue.cc:260] T 9ac64b6a72274b36a22727ae79227223 P b6ee352ce89c48d38b467042c4c3f9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.719501 21942 leader_election.cc:290] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 election: Requested vote from peers 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:33.721645 22056 ts_tablet_manager.cc:1428] T 9ac64b6a72274b36a22727ae79227223 P b6ee352ce89c48d38b467042c4c3f9fc: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20250111 02:58:33.731667 21700 raft_consensus.cc:2463] T dacdeedd62a944819edbf93c9f57d269 P 6830bca1186e4084868808884170cc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6ee352ce89c48d38b467042c4c3f9fc in term 1.
I20250111 02:58:33.733127 21492 leader_election.cc:304] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6830bca1186e4084868808884170cc33, b6ee352ce89c48d38b467042c4c3f9fc; no voters: 2754981a32a44bf889c5d2a226e013da
I20250111 02:58:33.734313 21942 raft_consensus.cc:2798] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:58:33.737052 22052 tablet_bootstrap.cc:492] T dacdeedd62a944819edbf93c9f57d269 P 2754981a32a44bf889c5d2a226e013da: No bootstrap required, opened a new log
I20250111 02:58:33.737520 22052 ts_tablet_manager.cc:1397] T dacdeedd62a944819edbf93c9f57d269 P 2754981a32a44bf889c5d2a226e013da: Time spent bootstrapping tablet: real 0.078s user 0.011s sys 0.000s
I20250111 02:58:33.735635 21942 raft_consensus.cc:695] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [term 1 LEADER]: Becoming Leader. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Running, Role: LEADER
I20250111 02:58:33.738838 21942 consensus_queue.cc:237] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.740134 22052 raft_consensus.cc:357] T dacdeedd62a944819edbf93c9f57d269 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.740885 22052 raft_consensus.cc:383] T dacdeedd62a944819edbf93c9f57d269 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:33.741148 22052 raft_consensus.cc:738] T dacdeedd62a944819edbf93c9f57d269 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Initialized, Role: FOLLOWER
I20250111 02:58:33.750306 22052 consensus_queue.cc:260] T dacdeedd62a944819edbf93c9f57d269 P 2754981a32a44bf889c5d2a226e013da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.756136 22052 ts_tablet_manager.cc:1428] T dacdeedd62a944819edbf93c9f57d269 P 2754981a32a44bf889c5d2a226e013da: Time spent starting tablet: real 0.018s user 0.000s sys 0.006s
I20250111 02:58:33.775365 22052 tablet_bootstrap.cc:492] T 9ac64b6a72274b36a22727ae79227223 P 2754981a32a44bf889c5d2a226e013da: Bootstrap starting.
I20250111 02:58:33.781770 22052 tablet_bootstrap.cc:654] T 9ac64b6a72274b36a22727ae79227223 P 2754981a32a44bf889c5d2a226e013da: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:33.797298 22052 tablet_bootstrap.cc:492] T 9ac64b6a72274b36a22727ae79227223 P 2754981a32a44bf889c5d2a226e013da: No bootstrap required, opened a new log
I20250111 02:58:33.797748 22052 ts_tablet_manager.cc:1397] T 9ac64b6a72274b36a22727ae79227223 P 2754981a32a44bf889c5d2a226e013da: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.002s
I20250111 02:58:33.800376 22052 raft_consensus.cc:357] T 9ac64b6a72274b36a22727ae79227223 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.801234 22052 raft_consensus.cc:383] T 9ac64b6a72274b36a22727ae79227223 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:33.801533 22052 raft_consensus.cc:738] T 9ac64b6a72274b36a22727ae79227223 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Initialized, Role: FOLLOWER
I20250111 02:58:33.802289 22052 consensus_queue.cc:260] T 9ac64b6a72274b36a22727ae79227223 P 2754981a32a44bf889c5d2a226e013da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.805058 22052 ts_tablet_manager.cc:1428] T 9ac64b6a72274b36a22727ae79227223 P 2754981a32a44bf889c5d2a226e013da: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250111 02:58:33.807837 22057 raft_consensus.cc:491] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:33.808341 22057 raft_consensus.cc:513] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.810321 22057 leader_election.cc:290] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021), b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313)
I20250111 02:58:33.811728 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac64b6a72274b36a22727ae79227223" candidate_uuid: "6830bca1186e4084868808884170cc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" is_pre_election: true
I20250111 02:58:33.811918 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac64b6a72274b36a22727ae79227223" candidate_uuid: "6830bca1186e4084868808884170cc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2754981a32a44bf889c5d2a226e013da" is_pre_election: true
I20250111 02:58:33.812425 21556 raft_consensus.cc:2463] T 9ac64b6a72274b36a22727ae79227223 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6830bca1186e4084868808884170cc33 in term 0.
I20250111 02:58:33.812605 21401 raft_consensus.cc:2463] T 9ac64b6a72274b36a22727ae79227223 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6830bca1186e4084868808884170cc33 in term 0.
I20250111 02:58:33.813870 21635 leader_election.cc:304] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6830bca1186e4084868808884170cc33, b6ee352ce89c48d38b467042c4c3f9fc; no voters:
I20250111 02:58:33.814782 22057 raft_consensus.cc:2798] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:58:33.815200 22057 raft_consensus.cc:491] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:58:33.815621 22057 raft_consensus.cc:3054] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:33.825987 22057 raft_consensus.cc:513] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.829084 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac64b6a72274b36a22727ae79227223" candidate_uuid: "6830bca1186e4084868808884170cc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2754981a32a44bf889c5d2a226e013da"
I20250111 02:58:33.829744 21401 raft_consensus.cc:3054] T 9ac64b6a72274b36a22727ae79227223 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:33.836671 21401 raft_consensus.cc:2463] T 9ac64b6a72274b36a22727ae79227223 P 2754981a32a44bf889c5d2a226e013da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6830bca1186e4084868808884170cc33 in term 1.
I20250111 02:58:33.838083 21634 leader_election.cc:304] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2754981a32a44bf889c5d2a226e013da, 6830bca1186e4084868808884170cc33; no voters:
I20250111 02:58:33.845479 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac64b6a72274b36a22727ae79227223" candidate_uuid: "6830bca1186e4084868808884170cc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ee352ce89c48d38b467042c4c3f9fc"
I20250111 02:58:33.846153 21556 raft_consensus.cc:3054] T 9ac64b6a72274b36a22727ae79227223 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:33.845710 22057 leader_election.cc:290] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [CANDIDATE]: Term 1 election: Requested vote from peers 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021), b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313)
I20250111 02:58:33.846870 22057 raft_consensus.cc:2798] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:58:33.848057 22057 raft_consensus.cc:695] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [term 1 LEADER]: Becoming Leader. State: Replica: 6830bca1186e4084868808884170cc33, State: Running, Role: LEADER
I20250111 02:58:33.849619 22057 consensus_queue.cc:237] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:33.852980 21556 raft_consensus.cc:2463] T 9ac64b6a72274b36a22727ae79227223 P b6ee352ce89c48d38b467042c4c3f9fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6830bca1186e4084868808884170cc33 in term 1.
W20250111 02:58:33.956151 21637 connection.cc:537] server connection from 127.0.0.1:44906 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:58:33.958860 21758 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.18.20.124:36707)
W20250111 02:58:33.961104 21758 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.20.124:36707)
W20250111 02:58:33.962366 21758 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.18.20.124:36707)
W20250111 02:58:33.965107 21758 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.18.20.124:36707)
W20250111 02:58:33.974004 21336 connection.cc:537] client connection to 127.18.20.124:36707 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:58:33.974025 21491 connection.cc:537] client connection to 127.18.20.124:36707 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:58:33.978389 21334 connection.cc:537] server connection from 127.0.0.1:36064 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:58:33.979401 21946 connection.cc:537] server connection from 127.0.0.1:46502 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:58:33.980489 21951 negotiation.cc:337] Failed RPC negotiation. Trace:
0111 02:58:33.975432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.20.124:36707 (local address 127.18.20.67:58729)
0111 02:58:33.976433 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0111 02:58:33.976479 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0111 02:58:33.976742 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0111 02:58:33.979632 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":348}
W20250111 02:58:33.981184 22025 negotiation.cc:337] Failed RPC negotiation. Trace:
0111 02:58:33.975794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.20.124:36707 (local address 127.18.20.66:60297)
0111 02:58:33.976760 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0111 02:58:33.976807 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0111 02:58:33.977108 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0111 02:58:33.980276 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":337}
W20250111 02:58:33.981571 22081 negotiation.cc:337] Failed RPC negotiation. Trace:
0111 02:58:33.968886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.20.124:36707 (local address 127.0.0.1:44280)
0111 02:58:33.977814 (+ 8928us) negotiation.cc:107] Waiting for socket to connect
0111 02:58:33.979808 (+ 1994us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":7895,"thread_start_us":7395,"threads_started":1}
W20250111 02:58:33.982220 22058 negotiation.cc:337] Failed RPC negotiation. Trace:
0111 02:58:33.976005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.20.124:36707 (local address 127.18.20.65:40715)
0111 02:58:33.977587 (+ 1582us) negotiation.cc:107] Waiting for socket to connect
0111 02:58:33.977634 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0111 02:58:33.977908 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0111 02:58:33.981337 (+ 3429us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":943}
I20250111 02:58:34.073839 22082 raft_consensus.cc:491] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ebb149f387f469fbb3731becb98426f)
I20250111 02:58:34.074268 22082 raft_consensus.cc:513] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:34.076540 22082 leader_election.cc:290] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:58:34.079937 21945 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 02:58:34.083657 21946 leader_election.cc:336] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627): Network error: Client connection negotiation failed: client connection to 127.18.20.126:34627: connect: Connection refused (error 111)
W20250111 02:58:34.084192 21945 leader_election.cc:336] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:34.084542 21945 leader_election.cc:304] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf3a224ebe0341df84e1d1d8fdfb0f21; no voters: 3476b1c08aec44598f1c203c2cbcf016, 7ebb149f387f469fbb3731becb98426f
I20250111 02:58:34.085192 22082 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250111 02:58:34.261513 22055 raft_consensus.cc:491] T dacdeedd62a944819edbf93c9f57d269 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:34.262054 22055 raft_consensus.cc:513] T dacdeedd62a944819edbf93c9f57d269 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:34.263973 22055 leader_election.cc:290] T dacdeedd62a944819edbf93c9f57d269 P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:34.265199 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacdeedd62a944819edbf93c9f57d269" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" is_pre_election: true
I20250111 02:58:34.266808 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacdeedd62a944819edbf93c9f57d269" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33" is_pre_election: true
I20250111 02:58:34.267701 21700 raft_consensus.cc:2388] T dacdeedd62a944819edbf93c9f57d269 P 6830bca1186e4084868808884170cc33 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2754981a32a44bf889c5d2a226e013da in current term 1: Already voted for candidate b6ee352ce89c48d38b467042c4c3f9fc in this term.
I20250111 02:58:34.268756 21337 leader_election.cc:304] T dacdeedd62a944819edbf93c9f57d269 P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2754981a32a44bf889c5d2a226e013da; no voters: 6830bca1186e4084868808884170cc33, b6ee352ce89c48d38b467042c4c3f9fc
I20250111 02:58:34.269665 22055 raft_consensus.cc:3054] T dacdeedd62a944819edbf93c9f57d269 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:34.276664 22055 raft_consensus.cc:2743] T dacdeedd62a944819edbf93c9f57d269 P 2754981a32a44bf889c5d2a226e013da [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:58:34.528429 22086 consensus_queue.cc:1035] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:58:34.554543 22086 consensus_queue.cc:1035] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:58:34.619838 22097 raft_consensus.cc:491] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ebb149f387f469fbb3731becb98426f)
I20250111 02:58:34.620292 22097 raft_consensus.cc:513] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:34.621939 22097 leader_election.cc:290] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:58:34.626883 21946 leader_election.cc:336] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627): Network error: Client connection negotiation failed: client connection to 127.18.20.126:34627: connect: Connection refused (error 111)
W20250111 02:58:34.628161 21945 leader_election.cc:336] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:34.628520 21945 leader_election.cc:304] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf3a224ebe0341df84e1d1d8fdfb0f21; no voters: 3476b1c08aec44598f1c203c2cbcf016, 7ebb149f387f469fbb3731becb98426f
I20250111 02:58:34.629150 22097 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250111 02:58:34.730125 22055 consensus_queue.cc:1035] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [LEADER]: Connected to new peer: Peer: permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:58:34.758487 22055 consensus_queue.cc:1035] T 0d09de52e0dd44f79b706c6aef6cdb42 P 2754981a32a44bf889c5d2a226e013da [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:58:34.865355 22087 consensus_queue.cc:1035] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:58:34.896508 22087 consensus_queue.cc:1035] T 9ac64b6a72274b36a22727ae79227223 P 6830bca1186e4084868808884170cc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:58:35.193133 22112 raft_consensus.cc:491] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ebb149f387f469fbb3731becb98426f)
I20250111 02:58:35.193550 22112 raft_consensus.cc:513] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:35.195024 22112 leader_election.cc:290] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:58:35.200359 21946 leader_election.cc:336] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627): Network error: Client connection negotiation failed: client connection to 127.18.20.126:34627: connect: Connection refused (error 111)
W20250111 02:58:35.201035 21945 leader_election.cc:336] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:35.201447 21945 leader_election.cc:304] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf3a224ebe0341df84e1d1d8fdfb0f21; no voters: 3476b1c08aec44598f1c203c2cbcf016, 7ebb149f387f469fbb3731becb98426f
I20250111 02:58:35.203080 22112 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250111 02:58:35.729278 18513 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
/tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.126:34627
--webserver_interface=127.18.20.126
--webserver_port=35247
--builtin_ntp_servers=127.18.20.84:35279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--hive_metastore_uris=thrift://127.0.0.1:41043
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250111 02:58:36.127584 22128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:58:36.128218 22128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:58:36.128698 22128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:58:36.162791 22128 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250111 02:58:36.163108 22128 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250111 02:58:36.163391 22128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 02:58:36.163707 22128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:58:36.164072 22128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 02:58:36.164373 22128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 02:58:36.213775 22128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.20.84:35279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:41043
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.20.126:34627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.20.126
--webserver_port=35247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4120
TSAN enabled
I20250111 02:58:36.215214 22128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:58:36.216934 22128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:58:36.234529 22143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:36.279330 22149 raft_consensus.cc:491] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ebb149f387f469fbb3731becb98426f)
I20250111 02:58:36.279739 22149 raft_consensus.cc:513] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:36.281375 22149 leader_election.cc:290] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:58:36.287119 21946 leader_election.cc:336] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627): Network error: Client connection negotiation failed: client connection to 127.18.20.126:34627: connect: Connection refused (error 111)
W20250111 02:58:36.288307 21945 leader_election.cc:336] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:36.289038 21945 leader_election.cc:304] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf3a224ebe0341df84e1d1d8fdfb0f21; no voters: 3476b1c08aec44598f1c203c2cbcf016, 7ebb149f387f469fbb3731becb98426f
I20250111 02:58:36.290114 22149 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250111 02:58:36.238198 22128 server_base.cc:1034] running on GCE node
W20250111 02:58:36.236855 22146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:58:36.235214 22144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:58:37.514250 22128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:58:37.517683 22128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:58:37.519204 22128 hybrid_clock.cc:648] HybridClock initialized: now 1736564317519155 us; error 62 us; skew 500 ppm
I20250111 02:58:37.520097 22128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:58:37.532581 22128 webserver.cc:458] Webserver started at http://127.18.20.126:35247/ using document root <none> and password file <none>
I20250111 02:58:37.533737 22128 fs_manager.cc:362] Metadata directory not provided
I20250111 02:58:37.533983 22128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:58:37.542484 22128 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250111 02:58:37.547513 22171 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:58:37.548730 22128 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250111 02:58:37.549105 22128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/data,/tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/wal
uuid: "3476b1c08aec44598f1c203c2cbcf016"
format_stamp: "Formatted at 2025-01-11 02:58:14 on dist-test-slave-09x0"
I20250111 02:58:37.551673 22128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:58:37.579245 21793 master_runner.cc:186] Error getting master registration for 127.18.20.126:34627: Network error: Client connection negotiation failed: client connection to 127.18.20.126:34627: connect: Connection refused (error 111),
I20250111 02:58:37.579624 21793 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:58:37.604758 22128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:58:37.606359 22128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:58:37.606860 22128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:58:37.694924 22128 rpc_server.cc:307] RPC server started. Bound to: 127.18.20.126:34627
I20250111 02:58:37.695122 22225 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.20.126:34627 every 8 connection(s)
I20250111 02:58:37.698287 22128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskY_5DOU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736564218214065-18513-0/minicluster-data/master-0/data/info.pb
I20250111 02:58:37.700021 18513 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskY_5DOU/build/tsan/bin/kudu as pid 22128
I20250111 02:58:37.721333 22226 sys_catalog.cc:263] Verifying existing consensus state
I20250111 02:58:37.739216 22226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Bootstrap starting.
I20250111 02:58:37.752355 21462 heartbeater.cc:346] Connected to a master server at 127.18.20.126:34627
I20250111 02:58:37.766124 22226 log.cc:826] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Log is configured to *not* fsync() on all Append() calls
I20250111 02:58:37.855180 22226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=16 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:58:37.856240 22226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: Bootstrap complete.
I20250111 02:58:37.883916 22226 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:37.886292 22226 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3476b1c08aec44598f1c203c2cbcf016, State: Initialized, Role: FOLLOWER
I20250111 02:58:37.887099 22226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 5.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:37.890100 22233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:37.890908 22233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:37.891433 22226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:58:37.949405 22248 catalog_manager.cc:1260] Loaded cluster ID: 3cbb96e3d31f437a849351b045304259
I20250111 02:58:37.949800 22248 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: loading cluster ID for follower catalog manager: success
I20250111 02:58:37.960048 22248 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: acquiring CA information for follower catalog manager: success
I20250111 02:58:37.968938 22248 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 02:58:37.995893 21603 heartbeater.cc:346] Connected to a master server at 127.18.20.126:34627
I20250111 02:58:37.993523 22128 master_runner.cc:186] Error getting master registration for 127.18.20.124:36707: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111),
I20250111 02:58:38.020507 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:58:38.022778 22190 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" instance_seqno: 1736564304600135) as {username='slave'} at 127.18.20.66:58061; Asking this server to re-register.
I20250111 02:58:38.025034 21603 heartbeater.cc:463] Registering TS with master...
I20250111 02:58:38.029947 22191 ts_manager.cc:194] Registered new tserver with Master: b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313)
I20250111 02:58:38.141674 21750 heartbeater.cc:346] Connected to a master server at 127.18.20.126:34627
I20250111 02:58:38.145996 22191 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6830bca1186e4084868808884170cc33" instance_seqno: 1736564306843221) as {username='slave'} at 127.18.20.67:46351; Asking this server to re-register.
I20250111 02:58:38.147468 21750 heartbeater.cc:463] Registering TS with master...
I20250111 02:58:38.150574 22191 ts_manager.cc:194] Registered new tserver with Master: 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:38.223838 22191 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2754981a32a44bf889c5d2a226e013da" instance_seqno: 1736564302339577) as {username='slave'} at 127.18.20.65:42555; Asking this server to re-register.
I20250111 02:58:38.225538 21462 heartbeater.cc:463] Registering TS with master...
I20250111 02:58:38.229557 22191 ts_manager.cc:194] Registered new tserver with Master: 2754981a32a44bf889c5d2a226e013da (127.18.20.65:34021)
I20250111 02:58:38.398555 22257 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:38.399052 22257 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:38.401753 22257 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
I20250111 02:58:38.402856 21971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3476b1c08aec44598f1c203c2cbcf016" candidate_term: 6 candidate_status { last_received { term: 5 index: 26 } } ignore_live_leader: false dest_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" is_pre_election: true
I20250111 02:58:38.403761 21971 raft_consensus.cc:2388] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3476b1c08aec44598f1c203c2cbcf016 in current term 6: Already voted for candidate 7ebb149f387f469fbb3731becb98426f in this term.
W20250111 02:58:38.403800 22175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 02:58:38.408272 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:38.408747 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:38.409677 22257 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 5 FOLLOWER]: Advancing to term 6
I20250111 02:58:38.419538 22257 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250111 02:58:38.792690 22260 raft_consensus.cc:491] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7ebb149f387f469fbb3731becb98426f)
I20250111 02:58:38.793234 22260 raft_consensus.cc:513] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:38.794873 22260 leader_election.cc:290] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:58:38.828477 21945 leader_election.cc:336] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:38.836372 22201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" candidate_term: 7 candidate_status { last_received { term: 6 index: 35 } } ignore_live_leader: false dest_uuid: "3476b1c08aec44598f1c203c2cbcf016" is_pre_election: true
I20250111 02:58:38.837329 22201 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf3a224ebe0341df84e1d1d8fdfb0f21 in term 6.
I20250111 02:58:38.838644 21946 leader_election.cc:304] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016, bf3a224ebe0341df84e1d1d8fdfb0f21; no voters: 7ebb149f387f469fbb3731becb98426f
I20250111 02:58:38.839550 22260 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250111 02:58:38.839905 22260 raft_consensus.cc:491] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7ebb149f387f469fbb3731becb98426f)
I20250111 02:58:38.840277 22260 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 6 FOLLOWER]: Advancing to term 7
I20250111 02:58:38.850615 22260 raft_consensus.cc:513] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:38.855475 22201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" candidate_term: 7 candidate_status { last_received { term: 6 index: 35 } } ignore_live_leader: false dest_uuid: "3476b1c08aec44598f1c203c2cbcf016"
I20250111 02:58:38.856083 22201 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 6 FOLLOWER]: Advancing to term 7
I20250111 02:58:38.863926 22201 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf3a224ebe0341df84e1d1d8fdfb0f21 in term 7.
I20250111 02:58:38.865257 21946 leader_election.cc:304] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016, bf3a224ebe0341df84e1d1d8fdfb0f21; no voters:
I20250111 02:58:38.872471 22260 leader_election.cc:290] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 election: Requested vote from peers 3476b1c08aec44598f1c203c2cbcf016 (127.18.20.126:34627), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:58:38.876117 21945 leader_election.cc:336] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:38.877329 22260 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 7 FOLLOWER]: Leader election won for term 7
I20250111 02:58:38.879210 22260 raft_consensus.cc:695] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [term 7 LEADER]: Becoming Leader. State: Replica: bf3a224ebe0341df84e1d1d8fdfb0f21, State: Running, Role: LEADER
I20250111 02:58:38.880236 22260 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 6.35, Last appended by leader: 35, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:38.885401 22267 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf3a224ebe0341df84e1d1d8fdfb0f21. Latest consensus state: current_term: 7 leader_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:38.886242 22267 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: This master's current role is: LEADER
I20250111 02:58:38.888615 22269 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:58:38.897799 22269 catalog_manager.cc:670] Loaded metadata for table default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f]
I20250111 02:58:38.899976 22269 catalog_manager.cc:670] Loaded metadata for table default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]
I20250111 02:58:38.901798 22269 catalog_manager.cc:670] Loaded metadata for table default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4]
I20250111 02:58:38.912290 22269 tablet_loader.cc:96] loaded metadata for tablet 9ac64b6a72274b36a22727ae79227223 (table default.table_46ab12e3d44441d982fda4c1aba20a2d [id=9089e2a62ec84b4b86a93cbc757190d4])
I20250111 02:58:38.914572 22269 tablet_loader.cc:96] loaded metadata for tablet a4a0a4fe6fdc4df08b38954165017d79 (table default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb])
I20250111 02:58:38.917938 22269 tablet_loader.cc:96] loaded metadata for tablet dacdeedd62a944819edbf93c9f57d269 (table default.table_836e7aa15e09458c9659cb4a8b7d352a [id=1e441333efa8493fac8f81eba63d7e7f])
I20250111 02:58:38.919589 22269 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:58:38.925377 22269 catalog_manager.cc:1260] Loaded cluster ID: 3cbb96e3d31f437a849351b045304259
I20250111 02:58:38.925698 22269 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:58:38.932375 22269 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:58:38.938282 22269 catalog_manager.cc:5910] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21: Loaded TSK: 0
I20250111 02:58:38.939718 22269 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:58:38.945426 22269 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 02:58:38.950898 22269 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250111 02:58:38.974417 22201 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Refusing update from remote peer bf3a224ebe0341df84e1d1d8fdfb0f21: Log matching property violated. Preceding OpId in replica: term: 5 index: 26. Preceding OpId from leader: term: 7 index: 37. (index mismatch)
W20250111 02:58:38.975404 21945 consensus_peers.cc:487] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 -> Peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Couldn't send request to peer 7ebb149f387f469fbb3731becb98426f. Status: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:58:38.976392 22260 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 26, Time since last communication: 0.000s
I20250111 02:58:38.980021 21961 master_service.cc:946] ReplaceTablet: received request to replace tablet b8cb51676b7346df96abe11be6d053cb from {username='slave'} at 127.0.0.1:46478
I20250111 02:58:38.984814 21960 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46478:
name: "default.Table_95e9bf8a21a4474781705bea1e0c7b91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:38.988085 21959 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46478:
name: "default.Table_b371b664c5b043be8b2f0353a9e57e8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:38.989648 21958 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46478:
table { table_name: "default.Table_15ebf4fa41db452c957f7aa50ddfc192" } new_table_name: "default.Table_7e8389e7a1b745a8898f6a0862a97327" modify_external_catalogs: true
I20250111 02:58:38.993940 21957 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46478:
table { table_name: "default.Table_46ab12e3d44441d982fda4c1aba20a2d" } new_table_name: "default.Table_77a9864685c248e9bb93b3b1f75adb6e" modify_external_catalogs: true
I20250111 02:58:38.995999 21956 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46478:
name: "default.Table_0939bec9a16f450fa14248fcc1cd2fc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:38.998028 21955 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46478:
table { table_name: "default.Table_836e7aa15e09458c9659cb4a8b7d352a" } modify_external_catalogs: true
I20250111 02:58:38.998996 21955 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46478:
table { table_name: "default.Table_836e7aa15e09458c9659cb4a8b7d352a" } modify_external_catalogs: true
I20250111 02:58:38.999178 21953 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46478:
name: "default.Table_dd8c5f570a5c46a4a461cb9d3c409afe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:58:39.000069 21954 master_service.cc:946] ReplaceTablet: received request to replace tablet b8cb51676b7346df96abe11be6d053cb from {username='slave'} at 127.0.0.1:46478
F20250111 02:58:39.003196 22015 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_95e9bf8a21a4474781705bea1e0c7b91
*** Check failure stack trace: ***
@ 0x7f16ac8f4950 google::LogMessage::Flush() at ??:0
@ 0x7f16ac8f46bf google::LogMessage::~LogMessage() at ??:0
I20250111 02:58:39.074280 22278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:39.077404 22278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:39.076082 22271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf3a224ebe0341df84e1d1d8fdfb0f21. Latest consensus state: current_term: 7 leader_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:39.078086 22271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:39.079903 22271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:39.080971 22271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:58:39.088917 22267 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } } }
I20250111 02:58:39.089792 22267 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3a224ebe0341df84e1d1d8fdfb0f21 [sys.catalog]: This master's current role is: LEADER
I20250111 02:58:39.110869 22276 mvcc.cc:204] Tried to move back new op lower bound from 7112967450491092992 to 7112967425322467328. Current Snapshot: MvccSnapshot[applied={T|T < 7112967450491092992 or (T in {7112967450491092992})}]
I20250111 02:58:39.111266 21533 tablet_service.cc:1467] Processing CreateTablet for tablet a4a0a4fe6fdc4df08b38954165017d79 (DEFAULT_TABLE table=default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:39.112541 21533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4a0a4fe6fdc4df08b38954165017d79. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f16b77ef82a kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250111 02:58:39.133899 21379 tablet_service.cc:1467] Processing CreateTablet for tablet a4a0a4fe6fdc4df08b38954165017d79 (DEFAULT_TABLE table=default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:39.135234 21379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4a0a4fe6fdc4df08b38954165017d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:39.179107 21681 tablet_service.cc:1467] Processing CreateTablet for tablet a4a0a4fe6fdc4df08b38954165017d79 (DEFAULT_TABLE table=default.table_15ebf4fa41db452c957f7aa50ddfc192 [id=22ba37c4c77d4f03981e24ae0ef1adcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:58:39.180711 21681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4a0a4fe6fdc4df08b38954165017d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:58:39.180932 22288 tablet_bootstrap.cc:492] T a4a0a4fe6fdc4df08b38954165017d79 P b6ee352ce89c48d38b467042c4c3f9fc: Bootstrap starting.
I20250111 02:58:39.181798 22290 tablet_bootstrap.cc:492] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da: Bootstrap starting.
I20250111 02:58:39.189865 22290 tablet_bootstrap.cc:654] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da: Neither blocks nor log segments found. Creating new log.
@ 0x7f16b77ef07c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250111 02:58:39.194619 22288 tablet_bootstrap.cc:654] T a4a0a4fe6fdc4df08b38954165017d79 P b6ee352ce89c48d38b467042c4c3f9fc: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:39.201887 22290 tablet_bootstrap.cc:492] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da: No bootstrap required, opened a new log
I20250111 02:58:39.202555 22290 ts_tablet_manager.cc:1397] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.005s
I20250111 02:58:39.205544 22290 raft_consensus.cc:357] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:39.206341 22290 raft_consensus.cc:383] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:39.206617 22290 raft_consensus.cc:738] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Initialized, Role: FOLLOWER
I20250111 02:58:39.207331 22290 consensus_queue.cc:260] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:39.219306 22288 tablet_bootstrap.cc:492] T a4a0a4fe6fdc4df08b38954165017d79 P b6ee352ce89c48d38b467042c4c3f9fc: No bootstrap required, opened a new log
I20250111 02:58:39.219780 22288 ts_tablet_manager.cc:1397] T a4a0a4fe6fdc4df08b38954165017d79 P b6ee352ce89c48d38b467042c4c3f9fc: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.000s
I20250111 02:58:39.222671 22288 raft_consensus.cc:357] T a4a0a4fe6fdc4df08b38954165017d79 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:39.223577 22288 raft_consensus.cc:383] T a4a0a4fe6fdc4df08b38954165017d79 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:39.223923 22288 raft_consensus.cc:738] T a4a0a4fe6fdc4df08b38954165017d79 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee352ce89c48d38b467042c4c3f9fc, State: Initialized, Role: FOLLOWER
@ 0x7f16b77f2d32 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250111 02:58:39.224699 22288 consensus_queue.cc:260] T a4a0a4fe6fdc4df08b38954165017d79 P b6ee352ce89c48d38b467042c4c3f9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:39.226467 21602 heartbeater.cc:502] Master 127.18.20.125:35029 was elected leader, sending a full tablet report...
I20250111 02:58:39.227303 22288 ts_tablet_manager.cc:1428] T a4a0a4fe6fdc4df08b38954165017d79 P b6ee352ce89c48d38b467042c4c3f9fc: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250111 02:58:39.229110 22294 tablet_bootstrap.cc:492] T a4a0a4fe6fdc4df08b38954165017d79 P 6830bca1186e4084868808884170cc33: Bootstrap starting.
W20250111 02:58:39.235097 21954 catalog_manager.cc:5359] Ignoring report from unknown tablet 0d09de52e0dd44f79b706c6aef6cdb42
W20250111 02:58:39.235404 21954 catalog_manager.cc:5359] Ignoring report from unknown tablet b8cb51676b7346df96abe11be6d053cb
I20250111 02:58:39.242821 22294 tablet_bootstrap.cc:654] T a4a0a4fe6fdc4df08b38954165017d79 P 6830bca1186e4084868808884170cc33: Neither blocks nor log segments found. Creating new log.
I20250111 02:58:39.246345 21952 master_service.cc:946] ReplaceTablet: received request to replace tablet b8cb51676b7346df96abe11be6d053cb from {username='slave'} at 127.0.0.1:46478
@ 0x7f16b77f2cea _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20250111 02:58:39.254258 21952 master_service.cc:946] ReplaceTablet: received request to replace tablet b8cb51676b7346df96abe11be6d053cb from {username='slave'} at 127.0.0.1:46478
I20250111 02:58:39.259483 21954 catalog_manager.cc:5526] T dacdeedd62a944819edbf93c9f57d269 P b6ee352ce89c48d38b467042c4c3f9fc reported cstate change: term changed from 0 to 1, leader changed from <none> to b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66). New cstate: current_term: 1 leader_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } health_report { overall_health: HEALTHY } } }
I20250111 02:58:39.262689 21954 catalog_manager.cc:5526] T 9ac64b6a72274b36a22727ae79227223 P b6ee352ce89c48d38b467042c4c3f9fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 6830bca1186e4084868808884170cc33 (127.18.20.67). New cstate: current_term: 1 leader_uuid: "6830bca1186e4084868808884170cc33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } } }
@ 0x7f16b77f2c7a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20250111 02:58:39.310070 22290 ts_tablet_manager.cc:1428] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da: Time spent starting tablet: real 0.107s user 0.000s sys 0.007s
I20250111 02:58:39.317258 21457 heartbeater.cc:502] Master 127.18.20.125:35029 was elected leader, sending a full tablet report...
@ 0x7f16b77f2c42 std::__1::__function::__alloc_func<>::operator()() at ??:0
W20250111 02:58:39.330861 21952 catalog_manager.cc:5359] Ignoring report from unknown tablet 0d09de52e0dd44f79b706c6aef6cdb42
W20250111 02:58:39.331298 21952 catalog_manager.cc:5359] Ignoring report from unknown tablet b8cb51676b7346df96abe11be6d053cb
@ 0x7f16b77f1f3e std::__1::__function::__func<>::operator()() at ??:0
@ 0x59ea05 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x59e83a std::__1::function<>::operator()() at ??:?
@ 0x7f16aeb398d7 kudu::Thread::SuperviseThread() at ??:0
@ 0x468a00 __tsan_thread_start_func at ??:?
I20250111 02:58:39.367674 22294 tablet_bootstrap.cc:492] T a4a0a4fe6fdc4df08b38954165017d79 P 6830bca1186e4084868808884170cc33: No bootstrap required, opened a new log
I20250111 02:58:39.369884 22294 ts_tablet_manager.cc:1397] T a4a0a4fe6fdc4df08b38954165017d79 P 6830bca1186e4084868808884170cc33: Time spent bootstrapping tablet: real 0.141s user 0.040s sys 0.080s
I20250111 02:58:39.372539 22294 raft_consensus.cc:357] T a4a0a4fe6fdc4df08b38954165017d79 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:39.373471 22294 raft_consensus.cc:383] T a4a0a4fe6fdc4df08b38954165017d79 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:58:39.373754 22294 raft_consensus.cc:738] T a4a0a4fe6fdc4df08b38954165017d79 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830bca1186e4084868808884170cc33, State: Initialized, Role: FOLLOWER
I20250111 02:58:39.376760 22294 consensus_queue.cc:260] T a4a0a4fe6fdc4df08b38954165017d79 P 6830bca1186e4084868808884170cc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:39.379762 22294 ts_tablet_manager.cc:1428] T a4a0a4fe6fdc4df08b38954165017d79 P 6830bca1186e4084868808884170cc33: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250111 02:58:39.382198 21747 heartbeater.cc:502] Master 127.18.20.125:35029 was elected leader, sending a full tablet report...
W20250111 02:58:39.389540 21954 catalog_manager.cc:5359] Ignoring report from unknown tablet 0d09de52e0dd44f79b706c6aef6cdb42
W20250111 02:58:39.389859 21954 catalog_manager.cc:5359] Ignoring report from unknown tablet b8cb51676b7346df96abe11be6d053cb
@ 0x7f16b01b86db start_thread at ??:0
@ 0x7f16ab59971f clone at ??:0
I20250111 02:58:39.540822 22156 raft_consensus.cc:491] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:58:39.541349 22156 raft_consensus.cc:513] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:39.543560 22156 leader_election.cc:290] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:39.544984 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a0a4fe6fdc4df08b38954165017d79" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" is_pre_election: true
I20250111 02:58:39.544911 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a0a4fe6fdc4df08b38954165017d79" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33" is_pre_election: true
I20250111 02:58:39.545630 21556 raft_consensus.cc:2463] T a4a0a4fe6fdc4df08b38954165017d79 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2754981a32a44bf889c5d2a226e013da in term 0.
I20250111 02:58:39.545675 21700 raft_consensus.cc:2463] T a4a0a4fe6fdc4df08b38954165017d79 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2754981a32a44bf889c5d2a226e013da in term 0.
I20250111 02:58:39.547030 21337 leader_election.cc:304] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2754981a32a44bf889c5d2a226e013da, 6830bca1186e4084868808884170cc33; no voters:
I20250111 02:58:39.548003 22156 raft_consensus.cc:2798] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:58:39.548359 22156 raft_consensus.cc:491] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:58:39.548712 22156 raft_consensus.cc:3054] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:39.556733 22156 raft_consensus.cc:513] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
I20250111 02:58:39.558800 22156 leader_election.cc:290] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 election: Requested vote from peers b6ee352ce89c48d38b467042c4c3f9fc (127.18.20.66:45313), 6830bca1186e4084868808884170cc33 (127.18.20.67:43789)
I20250111 02:58:39.559839 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a0a4fe6fdc4df08b38954165017d79" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6ee352ce89c48d38b467042c4c3f9fc"
I20250111 02:58:39.560458 21556 raft_consensus.cc:3054] T a4a0a4fe6fdc4df08b38954165017d79 P b6ee352ce89c48d38b467042c4c3f9fc [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:39.560714 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a0a4fe6fdc4df08b38954165017d79" candidate_uuid: "2754981a32a44bf889c5d2a226e013da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830bca1186e4084868808884170cc33"
I20250111 02:58:39.561276 21700 raft_consensus.cc:3054] T a4a0a4fe6fdc4df08b38954165017d79 P 6830bca1186e4084868808884170cc33 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:58:39.567497 21556 raft_consensus.cc:2463] T a4a0a4fe6fdc4df08b38954165017d79 P b6ee352ce89c48d38b467042c4c3f9fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2754981a32a44bf889c5d2a226e013da in term 1.
I20250111 02:58:39.568271 21700 raft_consensus.cc:2463] T a4a0a4fe6fdc4df08b38954165017d79 P 6830bca1186e4084868808884170cc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2754981a32a44bf889c5d2a226e013da in term 1.
I20250111 02:58:39.568845 21335 leader_election.cc:304] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2754981a32a44bf889c5d2a226e013da, b6ee352ce89c48d38b467042c4c3f9fc; no voters:
I20250111 02:58:39.570027 22156 raft_consensus.cc:2798] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:58:39.571312 22156 raft_consensus.cc:695] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [term 1 LEADER]: Becoming Leader. State: Replica: 2754981a32a44bf889c5d2a226e013da, State: Running, Role: LEADER
I20250111 02:58:39.572743 22156 consensus_queue.cc:237] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 } } peers { permanent_uuid: "2754981a32a44bf889c5d2a226e013da" member_type: VOTER last_known_addr { host: "127.18.20.65" port: 34021 } } peers { permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 } }
W20250111 02:58:39.635892 21334 connection.cc:537] client connection to 127.18.20.125:35029 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:58:39.641517 21756 connection.cc:537] client connection to 127.18.20.125:35029 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:58:39.646306 21756 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.18.20.125:35029)
W20250111 02:58:39.647374 21756 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.20.125:35029)
W20250111 02:58:39.648722 21756 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.18.20.125:35029)
W20250111 02:58:39.650197 21756 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.18.20.125:35029)
I20250111 02:58:39.797065 22310 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:39.797494 22310 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:39.799157 22310 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:58:39.809183 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:58:39.810429 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:39.810776 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:39.811421 22310 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 02:58:40.064957 18513 master-stress-test.cc:577] Killing test thread 1/10
I20250111 02:58:40.378017 22156 consensus_queue.cc:1035] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [LEADER]: Connected to new peer: Peer: permanent_uuid: "6830bca1186e4084868808884170cc33" member_type: VOTER last_known_addr { host: "127.18.20.67" port: 43789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:58:40.394474 22323 consensus_queue.cc:1035] T a4a0a4fe6fdc4df08b38954165017d79 P 2754981a32a44bf889c5d2a226e013da [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ee352ce89c48d38b467042c4c3f9fc" member_type: VOTER last_known_addr { host: "127.18.20.66" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250111 02:58:40.452502 22328 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:40.452900 22328 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:40.454308 22328 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:58:40.459193 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:58:40.459699 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:40.460026 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:40.460551 22328 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 02:58:41.258306 22337 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:41.258715 22337 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:41.260248 22337 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:58:41.265053 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:58:41.265471 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:41.265810 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:41.266427 22337 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 02:58:42.073354 22355 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:42.073776 22355 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:42.075459 22355 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:58:42.080197 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 02:58:42.080614 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 02:58:42.080952 22173 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:42.081542 22355 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 02:58:43.026554 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:58:43.026947 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:58:44.376986 22391 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:44.377458 22391 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:44.378968 22391 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:58:44.380935 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20250111 02:58:44.384285 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:58:44.384711 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:44.385071 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:44.385625 22391 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 02:58:45.560500 22417 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:45.560907 22417 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:45.562312 22417 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:58:45.569336 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:58:45.570344 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:58:45.570715 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:45.571501 22417 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 02:58:47.454800 22442 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:47.455405 22442 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:47.457247 22442 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:58:47.468309 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 02:58:47.468927 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 02:58:47.469283 22173 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:47.469928 22442 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 02:58:48.032665 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:58:48.033093 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:58:53.038177 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:58:53.038540 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:58:54.066563 22531 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:58:54.066984 22531 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:58:54.068598 22531 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:58:54.070680 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250111 02:58:54.073254 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 02:58:54.073668 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 02:58:54.074023 22173 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:58:54.074592 22531 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250111 02:58:54.889683 21597 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4ea0 after lost signal to thread 21467
W20250111 02:58:54.890707 21597 debug-util.cc:398] Leaking SignalData structure 0x7b08000e3e20 after lost signal to thread 21600
I20250111 02:58:58.044838 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:58:58.045300 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:59:00.206940 22619 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:59:00.207515 22619 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:59:00.209224 22619 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:59:00.211732 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 02:59:00.215293 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:59:00.215961 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:59:00.216313 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:59:00.217092 22619 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 02:59:03.054673 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:59:03.055104 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:59:08.061537 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:59:08.061977 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:59:13.067178 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:59:13.067584 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:59:14.857497 22778 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:59:14.858062 22778 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:59:14.859724 22778 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:59:14.861713 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 02:59:14.864863 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:59:14.865401 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:59:14.865727 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:59:14.866312 22778 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 02:59:18.073886 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:59:18.074362 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 02:59:21.398250 21742 debug-util.cc:398] Leaking SignalData structure 0x7b08000def40 after lost signal to thread 21615
W20250111 02:59:21.399338 21742 debug-util.cc:398] Leaking SignalData structure 0x7b08000d7b00 after lost signal to thread 21745
I20250111 02:59:23.080237 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:59:23.080624 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 02:59:27.303292 21601 heartbeater.cc:643] Failed to heartbeat to 127.18.20.124:36707 (116 consecutive failures): Network error: Failed to ping master at 127.18.20.124:36707: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 02:59:27.725800 21747 heartbeater.cc:643] Failed to heartbeat to 127.18.20.125:35029 (96 consecutive failures): Network error: Failed to ping master at 127.18.20.125:35029: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:59:27.761780 21456 heartbeater.cc:643] Failed to heartbeat to 127.18.20.124:36707 (116 consecutive failures): Network error: Failed to ping master at 127.18.20.124:36707: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:59:28.087010 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:59:28.087539 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:59:31.254925 22972 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:59:31.255532 22972 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:59:31.257795 22972 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:59:31.260013 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 02:59:31.263367 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:59:31.264040 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:59:31.264469 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:59:31.265237 22972 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 02:59:33.093407 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:59:33.093875 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:59:35.536684 23012 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:59:35.537200 23012 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:59:35.538651 23012 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:59:35.543254 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:59:35.543748 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:59:35.544087 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:59:35.544718 23012 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 02:59:38.100600 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:59:38.101181 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:59:43.107328 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:59:43.107820 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:59:46.741302 23127 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:59:46.741909 23127 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:59:46.743868 23127 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:59:46.746652 22175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250111 02:59:46.749833 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 02:59:46.750342 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 02:59:46.750679 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:59:46.751327 23127 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 02:59:48.113606 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:59:48.114154 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:59:53.120208 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:59:53.120678 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 02:59:54.408115 23218 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 02:59:54.408517 23218 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 02:59:54.410166 23218 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 02:59:54.412191 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 02:59:54.415175 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 02:59:54.415668 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 02:59:54.416052 22173 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 02:59:54.416978 23218 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 02:59:58.127010 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 02:59:58.127559 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:00:03.133221 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:00:03.133625 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 03:00:05.339512 22222 debug-util.cc:398] Leaking SignalData structure 0x7b08000b6be0 after lost signal to thread 22134
W20250111 03:00:05.340092 22222 debug-util.cc:398] Leaking SignalData structure 0x7b08000a24a0 after lost signal to thread 22225
I20250111 03:00:08.168303 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:00:08.168769 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:00:08.789407 23372 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:00:08.790012 23372 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:00:08.792024 23372 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:00:08.794170 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:00:08.797883 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:00:08.798447 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:00:08.798766 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:00:08.799482 23372 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:00:13.174858 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:00:13.175276 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:00:15.088303 23445 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:00:15.088829 23445 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:00:15.090826 23445 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:00:15.095361 22175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:00:15.098997 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 03:00:15.104509 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 03:00:15.105017 22173 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:00:15.105727 23445 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:00:18.181383 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:00:18.181849 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 03:00:20.860236 21452 debug-util.cc:398] Leaking SignalData structure 0x7b0800086940 after lost signal to thread 21301
W20250111 03:00:20.861837 21452 debug-util.cc:398] Leaking SignalData structure 0x7b0800042e40 after lost signal to thread 21455
I20250111 03:00:23.187129 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:00:23.187492 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:00:24.766973 23552 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:00:24.767534 23552 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:00:24.769276 23552 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:00:24.771229 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:00:24.774437 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:00:24.774981 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:00:24.775352 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:00:24.775995 23552 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250111 03:00:27.588552 21602 heartbeater.cc:643] Failed to heartbeat to 127.18.20.125:35029 (216 consecutive failures): Network error: Failed to ping master at 127.18.20.125:35029: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:00:28.126317 21747 heartbeater.cc:643] Failed to heartbeat to 127.18.20.125:35029 (216 consecutive failures): Network error: Failed to ping master at 127.18.20.125:35029: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:00:28.151634 21457 heartbeater.cc:643] Failed to heartbeat to 127.18.20.125:35029 (218 consecutive failures): Network error: Failed to ping master at 127.18.20.125:35029: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 03:00:28.193843 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:00:28.194294 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 03:00:31.366720 21742 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4b40 after lost signal to thread 21615
W20250111 03:00:31.367619 21742 debug-util.cc:398] Leaking SignalData structure 0x7b0800070e40 after lost signal to thread 21745
I20250111 03:00:33.199442 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:00:33.199795 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:00:36.205720 23700 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:00:36.206283 23700 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
W20250111 03:00:36.211406 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20250111 03:00:36.207965 23700 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:00:36.225555 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:00:36.226614 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:00:36.226948 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:00:36.227610 23700 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:00:37.926211 22178 maintenance_manager.cc:418] P 3476b1c08aec44598f1c203c2cbcf016: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033343
I20250111 03:00:38.011667 22177 maintenance_manager.cc:642] P 3476b1c08aec44598f1c203c2cbcf016: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.084s user 0.048s sys 0.008s Metrics: {"bytes_written":11782,"cfile_init":1,"drs_written":1,"lbm_read_time_us":227,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2294,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":20,"thread_start_us":451,"threads_started":1,"wal-append.queue_time_us":779}
I20250111 03:00:38.013974 22178 maintenance_manager.cc:418] P 3476b1c08aec44598f1c203c2cbcf016: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1186 bytes on disk
I20250111 03:00:38.015833 22177 maintenance_manager.cc:642] P 3476b1c08aec44598f1c203c2cbcf016: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":237,"lbm_reads_lt_1ms":4}
I20250111 03:00:38.205901 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:00:38.206324 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:00:42.368610 23794 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:00:42.369113 23794 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:00:42.370656 23794 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:00:42.372623 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:00:42.375633 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:00:42.376399 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:00:42.376729 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:00:42.377453 23794 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250111 03:00:43.130950 21597 debug-util.cc:398] Leaking SignalData structure 0x7b08000e2440 after lost signal to thread 21467
W20250111 03:00:43.131829 21597 debug-util.cc:398] Leaking SignalData structure 0x7b08000da920 after lost signal to thread 21600
I20250111 03:00:43.211635 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:00:43.211992 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:00:48.218160 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:00:48.218725 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 03:00:50.855624 21452 debug-util.cc:398] Leaking SignalData structure 0x7b0800046540 after lost signal to thread 21301
W20250111 03:00:50.856943 21452 debug-util.cc:398] Leaking SignalData structure 0x7b08000da540 after lost signal to thread 21455
I20250111 03:00:51.601944 23946 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:00:51.602499 23946 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:00:51.604275 23946 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:00:51.606503 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:00:51.609856 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 03:00:51.610330 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 03:00:51.610632 22173 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:00:51.611289 23946 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:00:52.545329 23966 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:00:52.545799 23966 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:00:52.547468 23966 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:00:52.552042 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:00:52.552423 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:00:52.552775 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:00:52.553390 23966 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:00:53.223671 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:00:53.224048 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 03:00:55.579562 21742 debug-util.cc:398] Leaking SignalData structure 0x7b08000c5740 after lost signal to thread 21615
W20250111 03:00:55.580991 21742 debug-util.cc:398] Leaking SignalData structure 0x7b08000d3560 after lost signal to thread 21745
I20250111 03:00:58.230801 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:00:58.231343 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:01:03.236827 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:01:03.237186 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:01:03.952670 24175 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:01:03.953294 24175 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:01:03.956005 24175 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:01:03.959039 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250111 03:01:03.962317 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:01:03.962811 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:01:03.963131 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:01:03.963788 24175 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:01:05.630141 24207 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:01:05.630599 24207 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:01:05.632272 24207 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:01:05.637473 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:01:05.637938 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:01:05.638236 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:01:05.638759 24207 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:01:08.243288 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:01:08.243757 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 03:01:12.255282 21597 debug-util.cc:398] Leaking SignalData structure 0x7b0800044940 after lost signal to thread 21467
W20250111 03:01:12.256371 21597 debug-util.cc:398] Leaking SignalData structure 0x7b08000c6ca0 after lost signal to thread 21600
I20250111 03:01:13.249256 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:01:13.249622 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:01:18.255817 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:01:18.256265 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:01:20.680698 24401 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:01:20.681274 24401 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:01:20.683472 24401 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:01:20.686090 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250111 03:01:20.689249 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 03:01:20.689647 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 03:01:20.689926 22173 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:01:20.690497 24401 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:01:23.261760 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:01:23.262135 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 03:01:27.958977 21602 heartbeater.cc:643] Failed to heartbeat to 127.18.20.125:35029 (336 consecutive failures): Network error: Failed to ping master at 127.18.20.125:35029: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 03:01:28.268947 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:01:28.269542 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 03:01:28.489050 21747 heartbeater.cc:643] Failed to heartbeat to 127.18.20.125:35029 (336 consecutive failures): Network error: Failed to ping master at 127.18.20.125:35029: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:01:28.559940 21456 heartbeater.cc:643] Failed to heartbeat to 127.18.20.124:36707 (356 consecutive failures): Network error: Failed to ping master at 127.18.20.124:36707: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:01:30.409124 24508 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:01:30.409763 24508 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:01:30.411754 24508 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:01:30.414151 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:01:30.417569 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 03:01:30.418105 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 03:01:30.418571 22173 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:01:30.419377 24508 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250111 03:01:32.493379 22222 debug-util.cc:398] Leaking SignalData structure 0x7b080008dc40 after lost signal to thread 22134
W20250111 03:01:32.493963 22222 debug-util.cc:398] Leaking SignalData structure 0x7b0800045440 after lost signal to thread 22225
I20250111 03:01:33.314523 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:01:33.314904 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:01:38.321769 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:01:38.322271 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:01:43.327874 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:01:43.328289 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:01:45.132359 24661 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:01:45.132907 24661 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:01:45.134792 24661 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:01:45.137660 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:01:45.141070 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:01:45.141677 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:01:45.141950 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:01:45.142545 24661 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:01:48.334911 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:01:48.335479 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:01:50.100533 24728 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:01:50.101058 24728 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:01:50.102959 24728 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:01:50.107789 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:01:50.108327 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:01:50.108632 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:01:50.109239 24728 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:01:53.340873 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:01:53.341264 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:01:57.388808 24820 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:01:57.389389 24820 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:01:57.391691 24820 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:01:57.396898 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250111 03:01:57.401229 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:01:57.402019 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:01:57.402360 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:01:57.403156 24820 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:01:58.347185 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:01:58.347664 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:02:03.353189 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:02:03.353667 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:02:08.360039 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:02:08.360503 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:02:09.282891 25027 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:02:09.283507 25027 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:02:09.285406 25027 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:02:09.289219 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:02:09.292644 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:02:09.293419 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:02:09.293737 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:02:09.294363 25027 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:02:13.366477 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:02:13.366973 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:02:18.373822 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:02:18.374279 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:02:23.380117 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:02:23.380501 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:02:26.498901 25339 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:02:26.499485 25339 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:02:26.502254 25339 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:02:26.504797 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:02:26.508090 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 03:02:26.508710 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 03:02:26.509109 22173 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:02:26.509989 25339 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250111 03:02:28.313593 21602 heartbeater.cc:643] Failed to heartbeat to 127.18.20.125:35029 (456 consecutive failures): Network error: Failed to ping master at 127.18.20.125:35029: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 03:02:28.386977 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:02:28.387511 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 03:02:28.846204 21747 heartbeater.cc:643] Failed to heartbeat to 127.18.20.125:35029 (456 consecutive failures): Network error: Failed to ping master at 127.18.20.125:35029: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:02:28.972745 21457 heartbeater.cc:643] Failed to heartbeat to 127.18.20.125:35029 (458 consecutive failures): Network error: Failed to ping master at 127.18.20.125:35029: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 03:02:33.025542 25445 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:02:33.026199 25445 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:02:33.028903 25445 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:02:33.035787 21452 debug-util.cc:398] Leaking SignalData structure 0x7b08000d13c0 after lost signal to thread 21301
W20250111 03:02:33.037381 21452 debug-util.cc:398] Leaking SignalData structure 0x7b08000871a0 after lost signal to thread 21455
W20250111 03:02:33.037257 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:02:33.040709 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 03:02:33.041314 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 03:02:33.041725 22173 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:02:33.042644 25445 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:02:33.392379 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:02:33.392760 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 03:02:37.960280 21597 debug-util.cc:398] Leaking SignalData structure 0x7b08000e5ee0 after lost signal to thread 21467
W20250111 03:02:37.961251 21597 debug-util.cc:398] Leaking SignalData structure 0x7b08000873e0 after lost signal to thread 21600
W20250111 03:02:38.308737 22222 debug-util.cc:398] Leaking SignalData structure 0x7b08000b7d20 after lost signal to thread 22134
W20250111 03:02:38.309420 22222 debug-util.cc:398] Leaking SignalData structure 0x7b08000c1960 after lost signal to thread 22225
I20250111 03:02:38.426498 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:02:38.426951 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 03:02:42.566746 21742 debug-util.cc:398] Leaking SignalData structure 0x7b08000df4c0 after lost signal to thread 21615
W20250111 03:02:42.567848 21742 debug-util.cc:398] Leaking SignalData structure 0x7b08000dc100 after lost signal to thread 21745
I20250111 03:02:43.432191 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:02:43.432544 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:02:48.075160 25650 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:02:48.075749 25650 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:02:48.078001 25650 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:02:48.080896 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:02:48.083873 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:02:48.084498 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:02:48.084784 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:02:48.085458 25650 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:02:48.438154 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:02:48.438560 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:02:53.443840 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:02:53.444186 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:02:56.662420 25737 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:02:56.663164 25737 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:02:56.665254 25737 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:02:56.668938 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:02:56.672526 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 03:02:56.673538 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 03:02:56.673868 22173 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:02:56.674474 25737 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:02:58.450417 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:02:58.450825 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:03:03.395340 25831 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:03:03.395849 25831 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:03:03.397509 25831 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:03:03.399587 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:03:03.402949 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:03:03.403455 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:03:03.403743 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:03:03.404343 25831 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:03:03.455844 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:03:03.456225 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:03:08.462994 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:03:08.463507 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 03:03:09.682602 21452 debug-util.cc:398] Leaking SignalData structure 0x7b08000d8a20 after lost signal to thread 21301
W20250111 03:03:09.683692 21452 debug-util.cc:398] Leaking SignalData structure 0x7b08000e7e80 after lost signal to thread 21455
I20250111 03:03:12.576397 25957 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:03:12.576928 25957 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:03:12.578723 25957 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:03:12.580996 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:03:12.584340 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
W20250111 03:03:12.584808 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 03:03:12.585173 22173 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:03:12.585774 25957 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:03:13.474027 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:03:13.474485 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:03:18.481613 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:03:18.482229 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 03:03:19.757522 22222 debug-util.cc:398] Leaking SignalData structure 0x7b0800054f40 after lost signal to thread 22134
W20250111 03:03:19.758085 22222 debug-util.cc:398] Leaking SignalData structure 0x7b08000455e0 after lost signal to thread 22225
I20250111 03:03:20.773023 26064 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:03:20.773604 26064 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:03:20.775468 26064 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:03:20.777709 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:03:20.781181 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:03:20.781993 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:03:20.782389 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:03:20.783020 26064 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:03:23.512105 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:03:23.512566 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 03:03:28.349846 21601 heartbeater.cc:643] Failed to heartbeat to 127.18.20.124:36707 (595 consecutive failures): Network error: Failed to ping master at 127.18.20.124:36707: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:03:28.519747 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:03:28.520537 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 03:03:28.847929 21747 heartbeater.cc:643] Failed to heartbeat to 127.18.20.125:35029 (575 consecutive failures): Network error: Failed to ping master at 127.18.20.125:35029: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:03:29.474979 21457 heartbeater.cc:643] Failed to heartbeat to 127.18.20.125:35029 (578 consecutive failures): Network error: Failed to ping master at 127.18.20.125:35029: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
I20250111 03:03:33.526572 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:03:33.526952 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:03:38.533644 22128 master_runner.cc:186] Error getting master registration for 127.18.20.125:35029: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111),
I20250111 03:03:38.534116 22128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 03:03:38.599038 26330 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bf3a224ebe0341df84e1d1d8fdfb0f21)
I20250111 03:03:38.599658 26330 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3476b1c08aec44598f1c203c2cbcf016" member_type: VOTER last_known_addr { host: "127.18.20.126" port: 34627 } } peers { permanent_uuid: "bf3a224ebe0341df84e1d1d8fdfb0f21" member_type: VOTER last_known_addr { host: "127.18.20.125" port: 35029 } } peers { permanent_uuid: "7ebb149f387f469fbb3731becb98426f" member_type: VOTER last_known_addr { host: "127.18.20.124" port: 36707 } }
I20250111 03:03:38.601969 26330 leader_election.cc:290] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029), 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707)
W20250111 03:03:38.603631 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 03:03:38.606665 22173 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bf3a224ebe0341df84e1d1d8fdfb0f21 (127.18.20.125:35029): Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:03:38.607255 22175 leader_election.cc:336] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7ebb149f387f469fbb3731becb98426f (127.18.20.124:36707): Network error: Client connection negotiation failed: client connection to 127.18.20.124:36707: connect: Connection refused (error 111)
I20250111 03:03:38.607548 22175 leader_election.cc:304] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3476b1c08aec44598f1c203c2cbcf016; no voters: 7ebb149f387f469fbb3731becb98426f, bf3a224ebe0341df84e1d1d8fdfb0f21
I20250111 03:03:38.608563 26330 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 3476b1c08aec44598f1c203c2cbcf016 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 03:03:39.483525 22178 maintenance_manager.cc:418] P 3476b1c08aec44598f1c203c2cbcf016: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1186 bytes on disk
I20250111 03:03:39.496361 22177 maintenance_manager.cc:642] P 3476b1c08aec44598f1c203c2cbcf016: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"dirs.queue_time_us":836,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":241,"thread_start_us":1014,"threads_started":2,"wal-append.queue_time_us":881}
W20250111 03:03:40.057689 21756 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:03:40.058187 21756 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:03:40.058451 21756 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
W20250111 03:03:40.058702 21756 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
F20250111 03:03:40.071098 21770 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1736564620 (unix time) try "date -d @1736564620" if you are using GNU date ***
F20250111 03:03:40.074744 21775 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800004851) received by PID 18513 (TID 0x7fe8443f8700) from PID 18513; stack trace: ***
@ 0x2dac70 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2dd153 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
F20250111 03:03:40.074744 21775 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)F20250111 03:03:40.098846 21767 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0939bec9a16f450fa14248fcc1cd2fc8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7fe858c67980 (unknown) at ??:0
@ 0x7fe853e80fb7 gsignal at ??:0
@ 0x7fe853e82921 abort at ??:0
@ 0x2e0c66 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7fe856f3bda4 google::LogMessage::SendToLog() at ??:0
@ 0x7fe856f3c950 google::LogMessage::Flush() at ??:0
@ 0x7fe856f41a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x35d5be kudu::MasterStressTest::DeleteTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x38abbe _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3840
@ 0x38aa71 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x38a1bb std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2dab30 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7fe858c5c6db start_thread at ??:0
F20250111 03:03:40.074744 21775 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)F20250111 03:03:40.098846 21767 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0939bec9a16f450fa14248fcc1cd2fc8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)F000F20250111 03:03:40.103881 21772 master-stress-test.cc:297] 000F20250111 03:03:40.093300 21764 master-stress-test.cc:297] 000F20250111 03:03:40.085414 21769 master-stress-test.cc:297] F2025 111 3: 3:40. 90775 21765 master-stress-test.cc:297] 2025 111 3: 3:40. 82136 21766 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_95e9bf8a21a4474781705bea1e0c7b91 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.18.20.126:34627,127.18.20.125:35029,127.18.20.124:36707, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.20.125:35029: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7fe853f6371f clone at ??:0